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EXECUTIVE  SUMMARY 

Human  Systems  Integration  (HSI)  is  a  collection  of  policies,  processes,  methods,  and  tools  that  are  applied 
in  the  system  acquisition  process  to  ensure  people  are  incorporated  into  new  systems  in  ways  that  provide 
for  their  health  and  safety  and  that  support  system  performance  goals.  The  United  States  Coast  Guard 
(USCG)  develops  many  types  of  products  including  aircraft,  cutters,  small  boats,  communications 
equipment,  surveillance  equipment,  software  suites,  weapons,  and  personal  protective  equipment.  Other 
government  agencies  have  found  great  benefits  in  mission  performance  and  cost  savings  by  including  HSI 
activities  all  along  the  development/acquisition  process.  The  USCG  has  become  interested  in  a  more 
methodical  and  integrated  approach  to  HSI  and  recently  stood  up  CG-1B3  as  its  Technical  Authority  for 
HSI.  The  technical  authority  will  need  to  oversee/execute  a  variety  of  HSI  activities  during  an  acquisition. 
There  are  a  number  of  HSI  tools  and  methods  that  can  be  employed  (by  the  USCG  and-or  its  contractors)  to 
perform  HSI  activities  and  analyses.  This  report  presents  a  survey  of  34  HSI  tools  and  methods. 

The  report  begins  with  an  overview  of  HSI  within  the  context  of  Coast  Guard  acquisition  processes.  We 
define  the  domains  of  HSI  and  discuss  the  importance  of  human  operators  as  components  of  complex 
systems.  We  also  discuss  the  importance  of  considering  work  content  and  variability  in  the  operational 
environment  as  determinants  of  both  the  likely  human  performance  envelope  and  the  system’s  adaptive 
requirements  during  operations. 

A  foundational  methodology  for  application  of  HSI  to  the  acquisition  and  system  engineering  process  then 
is  briefly  discussed.  We  identify  ten  system  development  processes  in  which  HSI  considerations  play 

prominent  roles,  and  which  are  critical  to  successful  system  development  outcomes.  These  processes  are 

* 

mapped  to  the  six  major  phases  of  Coast  Guard  system  acquisition.  Their  contributions  to  each  phase  are 
discussed. 

We  close  the  introductory  section  of  the  report  with  consideration  of  a  small  set  of  tools  (“starter  kit”) 
within  the  context  of  a  notional  acquisition  example.  The  purpose  of  this  example  is  to  illustrate  how  this 
tool  subset  can  be  used,  in  combination,  to  address  many  of  the  HSI  challenges  that  arise  during  the  design 
and  development  of  a  new  system.  This  starter  tool  kit  was  chosen  to  have  broad  applicability  across  many 
of  the  development  phases  identified  in  earlier  sections  of  the  report  (see  Table  ES-1,  below). 

HSI  tools  and  methodologies  falling  into  four  categories  are  then  surveyed  against  a  number  of  criteria  in 
the  areas  of  content,  communication,  trade-off  support,  and  traceability.  The  presentation  of  tools  follows  a 
discussion  of  the  USCG  acquisition  process,  HSI  domains  and  activities,  and  a  short  example  showing  how 
some  of  these  tools  could  be  used  to  improve  acquisition  and  systems  engineering  processes. 


In  November  2008,  the  Department  of  Homeland  Security  (DHS)  published  interim  Acquisition  Directive  102-01  (see 
https://dhsonline.dhs. gov/portal/ihtml/tracking/viewdoc2.ihtml?doid=4658001').  It  describes  a  number  of  changes  to  DHS 
acquisition  policy,  including  new  names  for  the  acquisition  phases:  Need;  Analyze/Select;  Obtain;  and 
Produce/Deploy/Support.  While  the  names  have  changed,  the  development  objectives  of  each  phase  remain  essentially  the 
same,  as  do  the  potential  contributions  of  HSI  to  each  phase. 
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Table  ES-1 .  Mapping  of  HSI  “starter  kit”  tools  to  CG  acquisition  phases. 


HSI 

Tool  /  Method 

USCG  Acquisition  Phase 

Project 

Identification 

Project 

Initiation 

Concept  & 
Technology 
Development 

Capability 
Development  & 
Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

Concept  mapping 

S 

S 

S 

Cognitive  Function 
Analysis  (CFA) 

S 

S 

S 

Information  and 
Functional  Flow 
Analysis  (IFF A) 

S 

S 

S 

IMPRINT  Pro 

S 

S 

Jack 

S 

V 

Job  Assessment 
Software  System 
(JASS) 

S 

S 

Advisor  3.5 

S 

S 

Technique  for 

Human  Error  Rate 
Prediction  (THERP) 

The  tools  included  in  this  survey  should  help  acquisition  managers,  program  managers,  and  system 
designers  in  two  important  ways.  First,  by  providing  ways  to  articulate  and  document  HSI  issues,  the  tools 
can  be  used  to  inform  the  overall  acquisition  process.  Second,  these  tools  support  examination  of  the  HSI 
issues  whose  resolution  is  crucial  to  system  performance  and  the  satisfaction  of  user  needs.  These  tools  will 
enhance  and  strengthen  Coast  Guard  system  acquisition  effectiveness,  efficiency,  and  operational  outcomes 
by  providing  designers  with  the  means  to: 

•  understand  the  requirements  associated  with  operations,  critical  decisions,  and  workflow; 

•  identify  and  properly  allocate  functions  between  human  operators  and  systems; 

•  understand  errors  and  their  likely  consequences;  and 

•  carry  out  the  trades  required  to  optimize  system  effectiveness. 
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1  INTRODUCTION 


The  United  States  Coast  Guard  (USCG)  uses  competitive  acquisition  contracts  to  develop  many  types  of 
products,  including  cutters,  small  boats,  aircraft,  communications  equipment,  surveillance  equipment, 
software  suites,  weapons,  and  personal  protective  equipment.  The  complexity  of  many  of  the  systems  being 
acquired  has  increased  dramatically,  due  in  some  measure  to  the  complexity  and  performance  demands  of 
the  USCG’s  new  homeland  security  role.  More  sophisticated  and  complex  systems  require  more 
sophisticated  acquisition  policies,  organizations,  personnel,  and  processes.  The  USCG  is  actively  working 
to  improve  all  of  these  aspects  of  their  acquisition  environment.  Human  systems  integration  (HSI)  has  been 
identified  as  an  acquisition  element  that  has  not  been  incorporated  into  the  process  as  efficiently  as  it  might. 
HSI  is  a  collection  of  policies,  processes,  methods,  and  tools  that  are  applied  in  the  system  acquisition 
process  to  ensure  people  are  incorporated  into  new  systems  in  ways  that  provide  for  their  health  and  safety 
and  support  system  performance  goals.  The  methodical  incorporation  of  HSI  into  a  system  acquisition  can 
result  in  substantial  improvements  in  mission  performance,  as  well  as  reduced  cost  (Rothblum,  2007). 

1.1  Overview  of  the  “HSI  for  Coast  Guard  Acquisitions”  Task 

In  October  2007,  USCG  Office  of  Human  Resources,  HSI  Staff  (CG-1B3),  was  made  the  Technical 
Authority  for  HSI  in  all  Coast  Guard  (CG)  acquisitions.  To  assist  both  CG-1B3  and  the  Acquisition 
Directorate  (CG-9),  the  USCG  Research  and  Development  Center  (RDC)  initiated  a  task  to  identify  HSI 
policies,  processes,  and  tools  which  could  improve  the  application  of  HSI  within  USCG  acquisitions.  The 
government-contractor  HSI  Team  created  the  following  products: 

1)  HSI  information  brochure:  HSI  is  a  relatively  new  concept  for  the  USCG  acquisition  community. 
Two  products  were  developed  to  educate  and  inform  the  community  about  HSI.  The  first  product  was  a 
brochure  (U.S.  Coast  Guard  R&D  Center,  2007,  unpublished)  that  summarized  the  goals,  areas  of 
interest  and  analysis,  processes,  tools,  and  benefits  of  HSI  to  overall  system  development  efforts.  The 
brochure  also  illustrated  how  HSI  can  contribute  to  every  phase  of  the  CG’s  Major  Acquisition  Process. 

2)  HSI  support  for  CONOPS  and  ORD  development:  The  second  product  was  an  annotated  briefing 
describing  how  HSI  can  be  applied  in  Concept  of  Operations  (CONOPS)  and  Operational  Requirements 
Document  (ORD)  development  (U.S.  Coast  Guard  R&D  Center,  2008b).  A  clear  and  effective 
statement  of  system  requirements  is  essential  to  the  development  of  mission-capable  systems.  The 
CONOPS  and  ORD  are  key  requirements  documents  for  expressing  the  CG’s  needs,  requirements,  and 
intended  use  of  a  new  system.  Consequently,  it  is  imperative  that  HSI  considerations  be  incorporated 
into  these  documents.  The  annotated  briefing  provides  an  overview  of  HSI,  discusses  HSI  activities  that 
can  be  performed  in  support  of  CONOPS  and  ORD  development,  and  provides  examples  of  methods 
and  tools. 

3)  HSI  programs  in  other  agencies:  Within  the  USCG  and  other  Government  organizations,  the 
acquisition  process  is  driven  by  formal  policy  documents  such  as  regulations,  directives,  and 
instructions.  Some  organizations  have  promulgated  specific  policy  in  an  effort  to  ensure  that  HSI  is 
routinely  and  consistently  incorporated  into  the  acquisition  process.  These  HSI-specific  policies  address 


The  task  order,  “ Human  Systems  Integration  for  Coast  Guard  Acquisitions was  performed  by  Science  Applications 
International  Corporation  (SAIC)  under  contract  number  HSCG32-05-D-R00010,  “U.S.  Coast  Guard  Research  and 
Development^  Center.  Technical^  Support  Services. .” 


Acquisition  Directorate 

Research  &  Development  Center 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


factors  such  as  how  HSI  organizations  are  structured,  how  HSI  activities  are  incorporated  into  larger 
system  acquisition  efforts,  and  how  the  services  define  and  implement  the  technical  authority  associated 
with  HSI.  The  HSI  Team  addressed  these  questions  by  interviewing  HSI  representatives  of  three 
Department  of  Defense  (DoD)  services  (Army,  Navy,  and  Air  Force)  and  the  National  Aeronautics  and 
Space  Administration  (NASA),  acquiring  and  analyzing  technical  instructions  and  other  documents 
related  to  HSI  definitions  and  program  implementation,  and  by  attending  an  inter-service  workshop 
devoted  to  this  topic.  The  HSI  Team  then  produced  an  annotated  briefing  that  documented  these 
agencies’  HSI  organization,  management,  technical  processes,  and  technical  authority  policies,  and  then 
made  HSI  policy  and  process  recommendations  for  similar  inclusion  of  HSI  into  the  USCG  acquisition 
process  (U.S.  Coast  Guard  R&D  Center,  2008a). 

4)  HSI  tools  survey  and  recommendations:  HSI  encompasses  a  number  of  technical  domains  and  a 
diverse  set  of  activities.  Methods  and  tools  have  been  developed  that  enhance  the  focus  and 
effectiveness  of  HSI  analyses,  design,  and  other  activities  and  reduce  the  effort  required.  The  HSI  Team 
identified  HSI  tools  that  can  be  used  to  enhance  USCG  acquisition  and  system  development  processes. 
The  tools  surveyed  included  both  (1)  processes  and  methodologies  and  (2)  software  products  that  would 
be  of  value  during  the  concept  design,  development,  and  evaluation  phases  of  USCG  acquisitions.  This 
report  is  the  product  of  that  analysis. 

1.2  The  Organization  of  the  Report 

The  body  of  this  report  provides  overview  information,  while  specific  information  on  the  tools  has  been 
included  in  the  appendices.  Section  2  discusses  what  HSI  is,  its  role  in  the  acquisition  process,  and  the 
benefits  that  accrue  from  applying  HSI.  Section  3  provides  introductory  material  on  HSI  tools:  where  they 
fit  within  the  acquisition  process;  plus  recommendations  for  an  HSI  tools  “starter  kit”  for  CG  applications. 

A  command  and  control  (C2)  system  acquisition  example  is  provided  to  show  how  the  HSI  “starter  kit” 
could  be  applied  throughout  the  acquisition  process.  Included  is  a  brief  discussion  of  human  performance 
modeling  as  a  specialized  class  of  HSI  tools  that  can  provide  quantitative,  performance-based  insights  into  a 
number  of  HSI  domains  across  multiple  acquisition  phases.  The  discussion  of  human  performance 
modeling  also  introduces  an  example  (presented  in  more  detail  in  Appendix  F)  of  how  the  technology  is 
applied.  Section  3  ends  with  a  summary  of  the  survey. 

Detailed  information  on  the  tools  survey  is  provided  in  the  appendices.  Appendix  A  describes  how  the  tool 
survey  was  performed,  including  the  criteria  used  for  the  selection  of  tools.  The  next  four  appendices 
present  the  surveys  of  tools  within  four  general  classes:  simulation  and  analysis  tools  for  CONOPS  and 
ORD  development  (Appendix  B);  tools  to  estimate  manning  and  personnel  qualifications  (Appendix  C); 
workload  and  situation  assessment  tools  (Appendix  D);  and  workstation  and  cockpit  design  tools  (Appendix 
E).  The  last  appendix  (Appendix  F)  describes  human  performance  modeling  and  how  it  can  be  applied  to 
CG  acquisitions,  using  the  Unmanned  Aerial  Systems  (UAS)  as  an  example. 

The  survey  of  HSI  tools  was  conducted  in  the  context  of  well  defined  and  generally-accepted  HSI  practice 
overlaid  on  the  USCG  acquisition  process.  Understanding  the  survey  methodology  and  results  requires 
some  familiarity  with  both  areas.  We  assume  that  readers  are  generally  familiar  with  the  USCG  acquisition 
process,  but  recognize  that  HSI  will  be  a  new  concept  to  many.  Consequently,  Section  2  provides  an 
overview  of  HSI  and  situates  HSI  activities  in  the  context  of  the  USCG  acquisition  process.  The  discussion 
of  the  tools  survey  then  follows. 
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2  OVERVIEW  OF  HSI  AND  THE  USCG  ACQUISITION  PROCESS 


As  depicted  in  Figure  1,  modem  systems  can  be  enormously  complex.  While  mission  needs  are  key  factors 
shaping  system  design,  there  are  other  important  factors  as  well.  Regulatory  or  legal  constraints  can 
establish  physical  or  other  operating  bounds  (e.g.,  airspace  restrictions  on  the  operation  of  unmanned  aerial 
vehicles).  The  operating  environment  itself  will  impose  constraints  and  requirements  that  must  be 
considered  in  system  design.  Operations  in  the  arctic,  for  example,  pose  requirements  for  operating  in 
extreme  cold  and  stormy  weather  conditions.  Drug  interdiction  missions  require  the  ability  to  apprehend 
armed  vessels  while  defending  the  ship.  Refugee  interdiction/assistance  requires  the  ability  to  conduct  both 
rescue  operations,  as  required,  as  well  as  detention  functions.  While  many  people  think  of  systems  in  terms 
of  a  ship  or  an  aircraft  (called  a  platform  in  Figure  1),  there  can  be  a  number  of  components  of  a  system. 
These  include  command  and  control  nodes  and  the  logistics  subsystem.  Logistics  includes  providing  the 
base  or  port  from  which  the  system  operates,  and  sustaining  system  operation  by  replenishing  consumables, 
maintaining  and  repairing  equipment,  and  providing  trained  personnel.  System  operating  concepts  are  a 
key,  but  sometimes  overlooked,  aspect  of  a  system.  Doctrine  and  tactics,  location  of  bases,  and  mission 
duration  have  impacts  on  how  a  system  is  employed  to  accomplish  a  mission  along  with  other  attributes 
such  as  the  number  of  people  needed  (e.g.,  multiple  crews  for  endurance  missions). 
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Figure  1.  Personnel  are  a  key  element  of  the  total  system. 


People  are  a  system  component  that  intersects  with  all  others.  They  operate  the  platform;  maintain  and 
replenish  it;  and  operate  the  C2  node  that  directs  and  tasks  the  platform.  The  number,  types,  jobs,  skills,  and 
abilities  of  personnel  required  in  a  system  can  vary  significantly.  Failure  of  individuals  and  teams  to 
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perform  their  duties  can  significantly  impact  system  and  mission  performance.  Because  people  are  so  key 
to  system  success,  disciplines,  processes,  methods,  and  tools  have  been  developed  to  help  incorporate 
people  into  systems  in  ways  that  enhance  system  and  mission  performance  and  system  and  personnel  safety 
and  health.  This  is  the  work  of  HSI. 

2.1  HSI  Defined 

Army  Regulation  602-2*  defines  HSI  as  “a  comprehensive  management  and  technical  strategy,  initiated 
early  in  the  acquisition  process,  to  ensure  that  human  performance,  the  burden  the  design  imposes  on 
manpower,  personnel,  and  training  (MPT),  and  safety  and  health  aspects  are  considered  throughout  the 
system  design  and  development  processes.”  The  Army  has  done  more  than  any  other  service  or 
organization  to  implement  HSI  as  a  formal,  policy-driven  set  of  activities  within  the  acquisition  process. 
Consequently,  Army  documents  are  the  source  of  much  HSI  thinking  being  applied  by  other  organizations. 

There  are  several  significant  elements  of  the  definition.  The  first  is  that  HSI  is  both  a  management  and  a 
technical  strategy.  It  is  a  management  strategy  because  it  organizes  HSI  activities  in  conjunction  with 
broader  system  management  processes  (e.g.,  requirements  development,  requirements  and  design  reviews, 
etc.).  It  is  a  technical  strategy  because  methods,  tools,  and  processes  are  applied  to  develop  knowledge 
about  human  performance  in  a  system.  This  knowledge  can  provide  insight  into  human  performance  issues 
that  need  to  be  resolved  so  the  system  can  meet  its  mission  performance  objectives. 

A  particularly  important  aspect  of  HSI  is  understanding  the  capabilities  and  limitations  of  the  different 
personnel  involved  in  a  system.  This  recognizes  that  while  people  can  be  a  critical  enabling  component  of  a 
system,  they  do  have  limits.  When  system  demands  exceed  the  capacity  of  personnel  to  meet  those 
demands,  system  performance  can  be  affected  negatively  (and  bad  things  can  happen).  Understanding  the 
limits  of  personnel  in  a  system  helps  us  design  systems  that  exploit  human  abilities  and  avoid  limitations. 

As  suggested  in  the  definition,  enabling  human  performance  in  the  system  is  a  key  goal  of  HSI.  There  are  a 
number  of  domains  within  which  human  performance  can  be  affected.  These  include  MPT,  safety  and 
health,  and  human  factors  engineering  (HFE).  Slightly  different  lists  of  domains  are  used  by  the  different 
DoD  services.  A  good  core  set  of  domains  are  described  briefly  below. 

•  Manpower:  The  number  of  personnel  (both  military  and  civilian)  required,  authorized,  and  potentially  available  to  train, 
operate,  maintain,  and  support  a  system. 

•  Personnel:  The  human  aptitudes,  skills,  and  capabilities  required  to  operate,  maintain,  and  support  a  system. 

•  Training:  The  instruction  and  resources  required  to  provide  personnel  with  requisite  knowledge,  skills,  and  abilities  to 
properly  operate,  maintain,  and  support  the  system. 

•  HFE:  The  comprehensive  integration  of  human  capabilities  and  limitations  into  system  definition,  design,  development,  and 
evaluation  to  promote  effective  human-machine  integration  for  optimal  total  system  performance. 

•  System  safety:  The  design  and  operational  characteristics  of  a  system  that  minimize  the  possibilities  for  accidents  or 
mishaps  caused  by  human  error  or  system  failure. 

•  Health  hazards:  The  systematic  application  of  biomedical  knowledge  to  identify,  assess,  and  minimize  health  hazards 
associated  with  the  system’s  operation,  maintenance,  repair  or  storage,  such  as:  acoustic  energy,  toxic  substances  (biological 
and  chemical),  oxygen  deficiency,  radiation  energy,  shock,  temperature  extremes,  trauma,  and  vibration. 
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•  Personnel  survivability:  The  characteristics  of  a  system  that  reduce  fratricide  as  well  as  reduce  detectability  of  personnel, 
prevent  attack  if  detected,  prevent  damage  if  attacked,  minimize  medical  injury  if  wounded  or  otherwise  injured,  and 
minimize  physical  and  mental  fatigue. 

While  the  domains  are  distinct,  they  are  not  independent.  It  is  important  to  understand  that  HSI  domains  are 
all  parts  of  the  same  whole:  human  performance  in  a  system.  There  is  interaction  across  the  domains.  For 
example,  a  personnel  strategy  that  assumes  personnel  with  higher  aptitudes  will  be  used  to  operate  a  system 
can  result  in  reduced  training  time  and  cost  because  the  personnel  will  acquire  job  proficiency  faster. 
Manpower  can  be  negatively  affected,  however,  because  there  are  fewer  high-aptitude  candidates  in  the 
general  population,  and  competition  is  stiff  for  those  people.  As  a  result,  it  might  be  impossible  to  acquire 
and  retain  enough  personnel  to  operate  the  system,  or  the  cost  of  those  personnel  might  be  too  high.  The 
point  here  is  that  HSI  domains  should  be  treated  as  an  integrated  trade  space  so  that  all  aspects  of  the  human 
component  of  the  larger  system  can  be  shaped  to  most  cost-effectively  address  system  requirements. 


2.2  Analysis  of  Work  Content  and  Environments:  The  Fundamental  Focus  of  HSI 

Simply  put,  the  role  of  HSI  is  to  understand  the  work  that  people  do  in  a  system  and  use  that  knowledge  to 
design  “people”-related  system  components  and  capabilities  that  help  ensure  mission  success  of  the 
complete  system.  Figure  2  provides  an  overview  of  how  HSI  professionals  accomplish  this.  It  begins  with 
a  description  of  the  larger  system.  This  can  be  a  high-level  description  like  a  CONOPS  or  something  more 
detailed  like  an  ORD.  In  Figure  2  (left  panel),  a  notional  mission  analysis  of  the  Maritime  Security  Cutter 
Medium  (WMSM)  is  used  as  a  point  of  departure.  HSI  professionals  would  review  the  WMSM  system 
analysis  to  identify  system  components  and  functions  that  explicitly  or  implicitly  require  people  to  operate 
or  support  (maintain,  repair)  them.  In  the  figure,  for  example,  the  need  for  a  helmsman  to  support  the 
steering  function  is  identified.  By  accumulating  these  “work  requirements”  across  the  system,  HSI 
professionals  construct  a  body  of  knowledge  that  describes  the  “people  work  content”  for  the  system. 

People  work  content  is  described  in  terms  of  the  functions  and  tasks  (work  elements)  that  people  perform 
(Figure  2,  center  box).  These  work  elements  are  further  described  in  terms  of  features  and  factors  such  as 
equipment  items  used  (points  of  interaction  with  the  platform),  broader  mission  processes  supported,  and  the 
environment  in  which  the  work  is  performed  (with  an  eye  toward  hazards  or  extreme  conditions). 


HSI  professionals  then  analyze  the  people  work  content  to  identify  important  attributes  of  the  work  that 
have  implications  for  the  different  HSI  domains  (Figure  2,  bottom  right  table).  Tasks,  for  example,  will  be 
evaluated  to  determine  how  often  they  must  be  performed;  the  nature  of  performance  demands  (e.g.,  how 
quickly  or  well  they  must  be  performed,  physical  strength  requirements,  etc.);  the  impact  of  task  failure  on 
mission  performance  (task  criticality);  the  content  of  the  skills  and  knowledge  that  enable  task  performance; 
the  aptitudes  (e.g.,  abstract  thinking,  math  ability,  language  skills)  needed  to  learn  and  perform  the  task;  and 
the  hazards  associated  with  task  performance  (in  terms  of  equipment,  materials,  threats,  weather,  etc.)  that 
can  affect  the  performer.  This  general  body  of  work  or  task  analysis  data  then  is  used  to  support  more 
detailed  assessments  within  the  individual  HSI  domains  (Figure  2,  right  center  table).  In  these  assessments, 
domain  experts  focus  on  work  or  task  attributes  that  are  most  relevant  to  that  domain  (for  example,  skills 
and  aptitudes  necessary  to  use  the  new  system).  These  assessments  identify  potential  issues  within  a 
domain  (e.g.,  insufficient  numbers  of  personnel  with  required  skills)  and  offer  solutions  and 
recommendations  for  resolving  them.  HSI  leadership  is  responsible  for  reviewing  results  across  domains, 
identifying  the  trade-offs  that  are  available  (e.g.,  changes  in  the  HFE  interface  design  may  accommodate 
less  skilled  personnel),  and  using  that  information  to  generate  requirements  for  the  complete  human 
component  of  the  system  and  to  participate  in  the  cross-component  trade-offs  that  occur  at  the  system  level. 
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Figure  2.  HSI  is  about  understanding  and  supporting  the  work  people  do  in  a  system  to  ensure  mission 
success. 


2.3  Integration  of  HSI  into  the  Acquisition  Process 

Specifying  HSI  requirements  is  only  one  of  a  number  of  ways  that  work  content  data  are  applied.  The  data 
are  used  throughout  the  acquisition  process.  Work  content  data  provide  the  basis  for  design,  selection,  and 
development  of  user-related  system  components.  These  include  user  interfaces  (controls  and  displays)  and 
work  spaces;  tools  and  aids;  protective  gear;  training  systems  and  capabilities;  and  operating  procedures, 
tactics,  and  guidelines.  Work  content  data  also  are  used  to  create  test  regimens  that  assess  the  performance 
of  user-related  system  components  during  developmental  and  operational  testing.  The  work  data  help 
testers  identify  critical  performances  that  need  to  be  assessed  and  fashion  measures  and  methods  to  conduct 
the  assessment.  One  of  the  important  functions  of  the  tools  discussed  later  in  this  report  is  to  help  HSI 
professionals  frame,  translate,  or  otherwise  process  work  data  to  obtain  insights  into  the  effective  design  of 
user  components. 

From  the  point  of  view  of  HSI,  there  is  a  “standard”  set  of  development  activities  that  are  carried  out  in 
support  of  system  development  (Haskins,  et  al.,  2007;  Meister  &  Enderwick,  2001;  Sheridan,  2002).  These 
are  shown  in  Table  1  and  defined  in  the  bulleted  list  that  follows. 

Table  1.  “Standard”  HSI  development  activities. 

•  Concept  definition  •  Performance,  workload,  and  training  estimation 

•  Requirements  analysis  •  Requirements  review 

•  Function  analysis  and  function  allocation  •  Personnel  selection 

•  Task  design  •  Training  development 

•  Interface  and  team  development _ •  Performance  assurance  (Test  &  Evaluation  (T&E)) 
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•  Concept  definition  is  a  process  of  specifying  the  operational  need  for  a  new  or  modified  system,  the  general  nature  of  the 
needed  system,  a  high-level  system  CONOPS,  and  operating  parameters  and  constraints  for  the  to-be-acquired  system. 

•  Requirements  analysis  is  a  process  of  reviewing,  assessing,  prioritizing,  and  balancing  the  needs  and  constraints  of  all 
stakeholders  pertinent  in  a  system  acquisition  effort;  defining  a  set  of  requirements  based  on  these  activities;  and 
transforming  those  requirements  into  functional  and  technical  descriptions  that  will  satisfy  stakeholder  needs.  Requirements 
analysis  typically  is  segmented  into  three  general  activities:  eliciting,  analyzing,  and  recording  requirements. 

•  Function  analysis  and  function  allocation  typically  take  place  as  coordinated  and  interdependent  activities.  They  involve 
examination  of  defined  functions  so  as  to  identify  all  sub-functions  necessary  to  accomplish  each  target  function.  Sub¬ 
functions  will  normally  be  arrayed  in  a  functional  architecture  to  show  their  relationships  to  one  another  and  to  other  system 
elements,  as  well  as  to  internal  and  external  interfaces.  Within  the  functional  architecture,  requirements  at  higher  levels  will 
often  flow  down  to  allocations  at  lower  levels.  Task  analysis,  a  typical  early  design  stage  activity  of  human  factors 
engineers,  is  a  form  of  function  analysis. 

•  Task  design  is  a  process  of  structuring  and  organizing  the  activities  of  human  operators  to  (1)  satisfy  the  operational 
requirements  of  the  system,  (2)  satisfy  the  constraints  present  in  the  system,  and  (3)  exhibit  adaptive  capabilities  in  the  face  of 
a  dynamic  environment. 

•  Interface  and  team  development  are  interdependent  activities,  since  the  interface  design  will  depend  on  the  needs  of  the 
team.  Interface  development  involves  creating  artifacts  and  procedures  that  allow  user  control  of  the  system;  provide  data, 
information,  and  other  feedback  about  system  state;  and  allow  manipulation,  analysis,  and  storage  of  newly  created  data  and 
knowledge  structures.  Team  development  involves  creating  artifacts  and  procedures  that  define  organizations  and  processes 
to  be  used  by  both  defined  and  ad-hoc  teams  when  carrying  out  operational  functions.  Both  interface  and  team  development 
represent  instantiations  of  the  foregoing  task  design. 

•  Performance,  workload,  and  training  estimation  are  analytical  processes  that  focus  on  estimating  the  levels  of 
performance,  effectiveness,  and  proficiency  attainable  with  the  task  designs,  interfaces,  and  team  processes  defined  in  the 
previous  stages,  discussed  above.  It  is  important  to  note  that  complete  estimates  of  performance  should  include  a 
consideration  of  both  ideal  and  degraded  environmental,  physical,  and  psychological  dimensions.  Measurements  should 
include  both  macro-scale  (e.g.,  decisions  made  by  commanders  and  operator  situation  awareness)  and  micro-scale  (e.g., 
recognition  accuracy,  tracking  accuracy,  and  reaction  times)  items. 

•  Requirements  review  is  the  process  of  ensuring  that  task  design,  interface  and  team  development,  and  performance, 
workload,  and  training  estimations  all  can  be  justified  against  the  system  and  subsystem  requirements  developed  in  the  early 
phases  of  system  acquisition.  In  essence,  the  goal  of  this  activity  is  to  develop  some  means  of  tracing  the  design 
commitments  of  later  phases  to  originating  and  system  requirements  of  earlier  phases.  This  is  a  challenge  in  and  of  itself, 
made  more  difficult  by  the  lack  of  methods  or  tools  in  either  the  systems  engineering  or  HSI  communities. 

•  Personnel  selection  involves  selecting  operators  and  other  personnel  based  on  the  knowledge,  skills,  and  abilities  -  along 
with  the  cognitive  and  physical  capabilities  -  required  to  fulfill  the  roles  that  each  person  will  play  in  interacting  with  the 
system.  These  roles  include  operating,  maintaining,  supporting,  and  managing  the  system  in  all  environments  and  operating 
conditions  envisioned  by  the  system  originators. 

•  Training  development  encompasses:  (1)  the  design,  specification,  and  implementation  of  the  instruction  and  resources  that 
(2)  will  be  required  to  bring  to,  and  maintain,  “proficiency”  of  (3)  those  personnel  who  will  interact  with  (operate,  maintain, 
support,  and  manage)  the  system  throughout  its  lifespan  and  to  maintain  that  proficiency  at  a  defined  level.  Note  that  a 
crucial  facet  of  training  development  includes  defining  “proficiency”  for  each  of  the  roles  just  mentioned. 

•  Performance  assurance  (T&E)  is  an  evaluative  process.  It  encompasses  all  tests  and  evaluations  that  will  be  carried  out 
over  the  course  of  system  development.  Developmental  T&E  (DT&E)  and  operational  T&E  (OT&E)  are  included  under  this 
activity,  as  are  earlier  and  less  formal  performance  assurance  activities  such  as  formative  evaluations  and  usability 
evaluations. 
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Each  of  these  activities  can  be  overlaid  on  the  Coast  Guard  acquisition  phases  in  the  manner  shown  in 

* 

Figure  3.  As  suggested  by  this  figure,  the  HSI  development  activities  are  both  overlapping  and  fairly 
loosely  defined  in  terms  of  their  extent  across  acquisition  phases.  The  figure  also  suggests  that  these 
activities  are  linear.  While  some  activities  will  tend  to  follow  others,  there  are  many  points  of  feedback  and 
other  interdependencies  between  activities.  Primarily,  this  is  because  outputs  of  earlier  activities  often  will 
serve  as  inputs  to  later  activities;  but,  as  mentioned  earlier,  some  activities  are  performed  in  parallel,  and  the 
interaction  between  them  is  important  to  their  maturation/optimization.  The  main  point  to  be  made  is  that 
the  range  of  HSI  activities  supports  most  acquisition  phases.  Most  of  these  activities  tend  to  cluster  in  the 
Concept  &  Technology  Development  (C&TD)  and  Capability  Development  &  Demonstration  (CD&D) 
phases.  However,  successful  application  of  HSI  in  these  phases  is  enabled  by  HSI  involvement  in  Project 
Initiation.  The  presence  of  tools  across  most  of  the  acquisition  phases  enables  a  consideration  of  HSI 
throughout  the  system  development. 

Within  the  context  of  a  specific  acquisition,  HSI  planning  activities  are  conducted  early  in  the  acquisition  to 
specify  the  processes,  methods,  and  tools  that  will  be  applied  within  the  different  HSI  development 
activities.  The  goal  here  is  to  fashion  an  approach  to  the  HSI  portion  of  the  acquisition  that  is  consistent 
with  factors  such  as  complexity,  mission  criticality,  and  risk  associated  with  the  system  being  acquired  and 
the  time  and  resources  available  for  the  HSI  activities.  To  be  most  effective,  HSI  planning  activities  should 
be  conducted  by  experienced  professionals  who  are  knowledgeable  of  the  broad  range  of  HSI  processes, 
methods,  and  tools  available.  Experienced  HSI  professionals  can  create  an  integrated  HSI  approach  that 
will  focus  on  the  most  critical  domains  and  human  performance  issues  and  will  maximize  the  reuse  of 
products  and  data  across  HSI  development  activities. 
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Figure  3.  HSI  development  activities  mapped  to  Coast  Guard  acquisition  phases. 


* 

In  November  2008,  the  Department  of  Homeland  Security  (DHS)  published  interim  Acquisition  Directive  102-01  (see 
https://dhsonhne.dhs.gov/portal/ihtml/trackingMewdoc2.jhtml?doid=4658001).  It  describes  a  number  of  changes  to  DHS 
acquisition  policy,  including  new  names  for  the  acquisition  phases:  Need;  Analyze/Select;  Obtain;  and 
Produce/Deploy/Support.  While  the  names  have  changed,  the  development  objectives  of  each  phase  remain  essentially  the 
same,  as  do  the  potential  contributions  of  HSI  to  each  phase. 
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2.4  HSI  Benefits 

Up  to  this  point,  the  discussion  has  been  more  about  what  HSI  is  and  what  it  does.  This  section  concludes 
the  overview  of  HSI  with  a  brief  discussion  of  the  benefits  of  HSI  to  acquisition  programs  that  apply  it. 

Some  of  the  key  benefits  include: 

•  Risk  reduction:  Perhaps  one  of  the  greatest  benefits  of  HSI  is  that  it  reduces  risk  associated  with  what  often  is  the  most 
expensive  and  complex  portion  of  a  system:  the  people.  By  helping  the  systems  engineering  team  identify  and  resolve 
people-related  risks  early  in  the  acquisition  process,  HSI  professionals  can  help  the  team  avoid  costly  surprises  and  re-designs 
later  in  the  program. 

•  Cost  management  and  reduction:  Cost  is  always  a  big  factor  in  system  acquisition.  By  establishing  clear,  human- 
component  requirements  early  in  the  acquisition  process,  HSI  professionals  help  the  acquisition  team  to  create  a  more 
realistic  trade  space  for  evaluating  system  alternatives  and  to  develop  more  realistic  expectations  of  system  cost.  HSI 
professionals  also  can  help  lower  system  acquisition  costs  by  avoiding  design  decisions  that  later  have  to  be  revisited  and 
corrected  and  by  developing  the  most  cost-effective  solutions  for  manning,  personnel  selection,  training,  and  safety.  In 
acquisitions  where  HSI  has  not  been  incorporated,  serious  design  flaws  can  result  which  significantly  degrade  human  and 
total  system  performance.  Correcting  these  deficiencies  late  in  the  development  can  incur  huge  costs,  as  high  as  a  factor  of 
100  times  the  original  cost  (Haskins,  et  al.,  2007,  based  on  a  Defense  Acquisition  University  statistical  analysis). 

•  Improved  mission  performance:  An  “intangible”  cost  of  not  properly  considering  HSI  factors  shows  up  in  the  loss  of 
effectiveness  and  efficiency  when  a  fielded  system  contains  errors  or  sub-optimal  interfaces  introduced  in  design.  This  has 
the  added  cost  of  increasing  overall  life-cycle  costs  because  of  the  extra  training  needed  to  attain  and  maintain  effective 
performance  levels.  In  some  cases,  poorly-designed  systems  create  so  much  user  frustration  that  they  lead  to  personnel 
attrition  -  offsetting  personnel  loss  can  be  a  huge  cost  to  the  organization.  By  helping  design  a  more  balanced,  effective 
integration  of  people,  hardware,  and  software,  HSI  professionals  help  ensure  the  system  meets  its  mission  performance 
objectives  in  a  cost-effective  manner. 

•  Safety  and  hazard  reduction:  Given  the  hazards  of  CG  missions  and  the  environments  within  which  CG  systems  operate, 
safety  is  a  particularly  important  consideration.  Safer  systems  not  only  mean  fewer  deaths  and  injuries  to  personnel,  it  also 
means  fewer  accidents  that  result  in  damage  to  CG  systems  and  equipment  and  to  property  in  their  operating  environment. 
Generally,  safety  issues  accrue  from  errors  in  design.  Many,  if  not  most,  accidents  evolve  from  a  long  chain  of  errors  and-or 
sub-optimal  system  behaviors  (i.e.,  “an  accident  waiting  to  happen”).  HSI  methods  and  tools  make  it  possible  to  discover 
these  causal  chains  at  an  early  stage  of  design  so  they  can  be  corrected,  or  at  least  rendered  non-catastrophic. 

A  previous  survey  on  the  uses  of  HSI  processes  in  the  DoD  services  indicated  that  a  number  of  cost  savings 
and  positive  returns  on  investment  accrue  to  those  who  integrate  human-system  considerations  in  their 
system  development  efforts.  One  of  the  most  important  advantages  is  in  the  area  of  cost  savings.  As  stated 
above,  allowing  design  “mistakes”  to  propagate  through  a  development  effort  and  into  the  fielded  system 
can  be  very  expensive  in  terms  of  (1)  corrections  to  hardware  and-or  software  (a  monetary  dimension), 

(2)  additional  and  more  frequent  training  required  to  achieve  and  maintain  proficiency  (a  time  dimension), 
and  (3)  degradation  of  system  performance  (a  performance  dimension).  Simply  stated,  cost  is 
multidimensional  and  often  intangible.  Much  of  the  cost  associated  with  a  system  will  be  associated  with  its 
human  components:  operators,  maintainers,  trainers,  and  so  on.  Opportunity  costs,  as  well  as  monetary 
costs,  are  relevant.  Including  human  considerations  early  in  development  as  an  integral  part  of  the  larger 
development  process  provides  the  opportunity  to  identify  and  eliminate  design  “mistakes”  before  they 
become  costly  in  one  or  more  of  the  dimensions  discussed  above. 

Achieving  cost  savings  through  HSI  can  be  realized  by  following  seven  general  tenets,  as  outlined  below 
(Haskins,  et  al.,  2007). 
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1)  Initiate  HSI  into  development  systems  engineering  early. 

2)  Identify  HSI  issues  and  plan  appropriate  analyses  that  are  program-specific  and  issue-oriented. 

3)  Inject  HSI  considerations  into  system  requirements  development. 

4)  Make  HSI  a  factor  in  source  selection. 

5)  Execute  integrated  HSI  technical  processes  throughout  the  program  by  developing  an  HSI  integrated 
architecture  and  including  HSI  considerations  in  each  planning/engineering  document. 

6)  Conduct  proactive  HSI  trade-off  analyses. 

7)  Conduct  HSI  assessments  throughout  the  development  process. 

These  tenets,  based  on  the  experience  of  DoD  service-specific  acquisition  programs,  underlie  a  number  of 
system  development  success  stories.  Conversely,  when  these  HSI  tenets  are  not  applied  properly,  it 
generally  leads  to  costly  systems  which  fail  to  attain  the  hoped-for  level  of  mission  performance.  Table  2 
summarizes  a  few  of  the  success  stories  and  illustrates  the  diverse  ways  that  HSI  can  positively  affect 
program  and  system  success  factors. 
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Table  2.  Some  HSI  success  stories. 


Maintenance  Analysis  for  Joint  Biological  Point  Detection  Systems  (JBPDS).  Army  Manpower  &  Personnel 
Integration  (MANPRINT)  analysts  determined  the  operator  and  direct  support-level  maintenance  manpower 
requirements  and  spare  line  replaceable  units  (LRUs)  needed  for  the  JBPDS.  A  comparison  was  performed 
between  the  cost  of  standard  army  maintenance  using  35F  Special  Electronics  Repair  personnel  and  Contractor 
Logistics  Support  (CLS).  Results  demonstrated  a  $270k  savings  per  year  per  biological  detection  company  for  CLS 
for  wartime  requirements. _ 

Mine  Clearing  Training  for  Improved  Detection.  Army  training  experts  developed  a  specific  training  regimen  with 
the  AN/PSS-12  Hand-held  Mine  Detector  to  transfer  expert  detection  skills  to  soldiers;  designed  a  training  site  that 
could  efficiently  enable  soldiers  to  develop  the  required  skills  (and  could  be  practically  constructed  by  field  units); 
and  validated  low-metal  mine  simulants  for  use  in  training.  Results  of  tests  employing  these  improvements  showed 
significant  increases  in  the  probability  of  detection  in  relatively  little  training  time.  Repeatedly,  results  showed 
increases  in  probability  of  detection  from  less  than  20%  using  standard  techniques  to  75%  (less  than  1  hour  of 
training)  to  98%  (about  15  hours  of  training)  using  the  expert  techniques. _ 

Hand-held  GPS  Receiver  Operator  Performance.  Army  MANPRINT  experts  evaluated  dismounted  soldiers  using 
the  Defense  Advanced  GPS  Receiver  (DAGR)  in  the  field  and  observed  the  presence  of  a  fratricide  issue:  38%  of 
the  soldiers  (6  out  of  16)  incorrectly  reported  their  present  position  rather  than  the  target’s  position  during  a 
simulated  Call  for  Fire  Scenario.  The  MANPRINT  experts  recommended  use  of  a  pop-up  warning  message  (which 
was  incorporated  by  the  contractor)  and,  in  the  retest,  none  of  the  soldiers  (13  out  of  13)  incorrectly  reported  their 
present  position. _ 

C-17  Cargo  Aircraft.  Air  transportability  and  loading  issues  were  resolved  for  joint  delivery,  complex  ground 
refueling  operations,  the  number  of  people  required  for  complex  operations,  training  time,  down  time/maintenance 
time,  and  reduction  of  number  of  flight  crew  positions.  Crew  systems,  human  factors,  manpower,  personnel,  and 
training  efforts  were  credited  with  saving  2916  manpower  positions  over  predecessor  systems.  Life-cycle  cost 
savings  were  estimated  in  excess  of  $300M. _ 

F-22  RAPTOR.  Human  performance  HSI  challenges  for  the  F-22  included  the  management  of  manpower 
allocations  and  specialty  codes,  maintenance  manpower  compressions,  repair  of  low  observable  coatings,  servicing 
advance  oxygen  generating  systems,  control  and  display  integration,  and  net-centric  warfare  display  integration.  An 
integrated  training  system  also  was  required.  Development  of  this  aircraft  involved  a  major  shift  from  system  status 
orientation  to  tactical  warfighter-centric  displays.  The  Program  Manager  elevated  eight  Crew  Systems  and  HSI- 
related  issues  to  the  Work  Breakdown  Structure  Ill-level  for  the  first  time  ever.  A  detailed  MPT  analysis  ($2M 
investment)  led  to  $700M  in  life-cycle  cost  avoidance.  The  eventual  manpower  implementation  for  the  F-22  (2005- 
2006)  has  been  credited  with  $3B  in  life-cycle  cost  savings. _ 
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3  HSI  TOOLS  SURVEY 


As  noted  above,  HSI  covers  a  diverse  set  of  human  performance-related  domains  and  development 
activities.  Over  time,  HSI  researchers  and  practitioners  have  created  methods  and  tools  that  support  the 
conduct  of  key  activities.  A  wide  variety  of  tools  exist,  potentially  allowing  HSI  analysts  and  designers  to 
contribute  to  many  phases  of  development.  However,  because  of  the  diversity  of  the  available  tools,  it  can 
be  daunting  for  those  involved  in  system  development  to  locate  and  select  tools  for  optimal  application  at 
each  phase,  or  during  specific  processes,  of  a  development  effort.  The  existence  of  a  tool  compendium 
should  make  the  selection  and  acquisition  of  the  “best  tool  for  today’s  need”  more  tractable.  Through  this 
report,  we  hope  to  provide  the  HSI  Technical  Authority  (CG-1B3)  and  other  members  of  the  USCG 
acquisition  community  with  a  categorized  list  of  resources,  indexed  according  to  a  tool’s  relevance  and 
contribution  to  system  development.  By  using  this  off-the-shelf  resource,  one  can  quickly  locate  the  right 
tool  for  the  right  task  during  each  phase  of  development.  A  basic  question  that  can  be  asked  at  every  phase 
of  system  development  is  “are  we  doing  the  right  things,  and  are  we  doing  things  right?”  With  respect  to 
HSI,  the  existence  of  a  tools  survey  helps  address  this  question. 

All  of  the  tools  in  the  survey  are  available  as  either  commercial  off-the-shelf  (COTS)  or  Government  off- 
the-shelf  (GOTS).  The  goal  of  the  survey  was  to  identify  useful  HSI  tools  that  can  either  be  used  by  the  CG 
or  can  be  required  of  acquisition  contractors  during  system  development  efforts.  Contractors  would  use 
these  tools,  in  conjunction  with  other  methods  and  processes,  to  conduct  HSI  analyses  for  injection  into 
system  development  processes. 

The  HSI  tools  survey  presented  in  this  report  is  not  exhaustive.  All  available  tools  and  methodologies  have 
not  been  included  in  the  survey.  We  included  only  those  tools  that  were  relevant  to  development  in  a  Coast 
Guard  environment  and  that  were  easily  available  either  commercially  or  through  DoD  or  other  Government 
channels.  We  have  included  contact  information  for  each  tool  included  in  the  survey  to  simplify  tool 
acquisition.  Similarly,  the  surveys  of  individual  tools  are  not  exhaustive.  Each  survey  contains  only  the 
information  that  the  reviewers  felt  was  relevant  to  the  needs  of  the  Coast  Guard  acquisition  process.  This 
information  has  been  organized  into  four  broad  categories,  each  of  which  will  be  discussed  in  detail  in 
Appendix  A.  Briefly,  these  categories  address:  (1)  the  content  produced  by  each  tool;  (2)  the  means  by 
which  the  tool  communicates  its  output  to  the  larger  system  development  process;  (3)  methods  by  which  the 
tool  can  be  used  to  estimate  and  predict  human  performance  “errors”  and  conduct  trade-off  analyses  within 
and  across  HSI  domains;  and  (4)  the  degree  to  which  each  tool  provides  traceability  between  originating 
requirements  and  design  commitments. 


Many  HSI  tools  tend  to  be  domain-specific,  which  can  be  a  limitation.  As  discussed  in  Section  2.1,  HSI 
consists  of  the  domains  of  manpower,  personnel,  training,  HFE,  system  safety,  health  hazards,  and 
personnel  survivability.  The  issues  considered  by  each  domain  often  will  serve  as  drivers  of  acquisition- 
related  research  and  analysis  efforts.  It  is  also  necessary  to  manage  trade-offs  and  risks  among  the  different 
HSI  domains,  as  well  as  between  HSI  and  other  engineering  disciplines.  Consider,  for  example,  a  system 
designed  for  expert  users,  which  therefore  has  stringent  selection  criteria  for  specific  knowledge  and  skills 
(drawn  from  the  HFE  domain  (Folds,  2007)).  This  issue  leads  to  a  need  to  trade  off  human  factors 
considerations  (typically  focused  on  performance)  against  personnel  qualifications  (typically  focused  on 
aptitudes  and  preparation)  and  manpower  (the  number  of  people  available  with  the  required  qualifications). 
Trade-offs  against  other  aspects  of  the  system  design  will  be  needed  (perhaps  the  requirement  for  an  expert 
user  came  from  a  COTS  component  that  was  less  expensive  than  a  custom  design  with  a  simpler  user 
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interface  -  need  to  trade  off  component  costs  against  costs  of  training  and  increased  manpower).  Ideally, 
tools  for  HSI  should  allow  these  trades  to  be  made.  Few,  however,  support  this  ability. 

Table  3  lists  the  tools  surveyed  and  summarizes  the  distribution  of  HSI  domain  coverage  for  the  tools.  Tool 
names  are  shown  at  the  left  of  the  table  and  HSI  domains  are  shown  across  the  top.  Table  cells  contain  both 
a  letter  code  and  a  number  code.  The  letter  code  indicates  the  acquisition  phase  within  which  the  tool  can 
best  be  applied.  These  are  the  acquisition  phases  used  in  the  USCG  acquisition  process.  The  number  code 
indicates  the  survey  category,  taken  from  the  statement  of  work,  to  which  each  tool  applies.  (Note: 

Because  other  CG  organizations  already  deal  with  the  issues  of  safety,  survivability,  and  habitability,  these 
were  not  focus  areas  for  the  tools  survey.)  An  additional  code  following  each  tool  name  designates  tool 
type  and  breadth  of  applicability  across  HSI  domains.  Each  tool  was  categorized  as  one  of  three  types: 
software,  analysis  technique,  or  methodology.  Software  tools  are  self-explanatory.  Analysis  techniques 
provide  specific,  stepwise  procedures  for  exploration  and  analysis  of  particular  topics  relevant  to  system 
development.  They  usually  make  explicit  assumptions  about  the  nature  of  the  issue  under  analysis  (e.g., 
errors  arise  probabilistically  and  are  causally  associated,  or  strongly  correlated,  with  system  failures). 
Technique  of  Human  Error  Rate  Prediction  (THERP)  is  an  example  of  an  analysis  technique. 

Methodologies  provide  broad  ways  of  carrying  out  analyses  in  support  of  system  development;  however, 
they  are  neither  as  specific  nor  do  they  provide  the  programmed,  stepwise  procedures  that  analytical 
techniques  do.  They  might  be  based  on  theoretical  positions  but  these  often  are  high-level  “world  views” 
rather  than  specific,  rigorous  theoretical  statements.  Therefore,  they  leave  much  of  the  specific  technique  up 
to  the  individual  analyst,  subject  to  an  individual  analysis  problem.  Most  of  the  cognitive  systems 
engineering  methods  fall  into  this  category.  Each  tool  was  also  categorized  according  to  the  number  of  HSI 
domains  it  addressed.  Tools  that  addressed  only  one  or  two  HSI  domains  were  considered  “restricted;” 
those  that  addressed  three  or  more  domains  were  considered  “broad.”  For  example,  Improved  Performance 
Research  Integration  Tool  (IMPRINT)  is  considered  to  be  a  software  tool  with  broad  applicability  (code  = 
SB)  because  it  can  be  used  to  address  questions  in  four  HSI  domains,  whereas  Advisor  3.5  is  a  software  tool 
with  restricted  applicability  (code  =  SR)  because  it  applies  to  a  single  HSI  domain. 

As  can  be  seen  in  Table  3,  all  domains  received  coverage  across  the  tools  surveyed.  However,  this  coverage 
was  not  uniform.  Due  to  the  focus  of  this  project,  four  of  the  seven  HSI  domains  received  the  most 
coverage  from  the  tool  set  included  in  this  survey.  These  four  were  HFE,  training,  manpower,  and 
personnel.  All  34  tools  address  at  least  one  of  these  four  domains,  and  most  address  two  or  more  domains. 
Ten  of  the  tools  also  address  one  or  more  of  the  remaining  three  domains  (safety,  survivability,  and 
habitability).  About  two-thirds  of  the  tools  in  the  survey  are  software  applications;  the  rest  are  specific 
techniques  and  methodologies.  About  half  of  the  tools  are  general-purpose  tools,  while  the  others  are  more 
specialized. 

Full  surveys  of  each  tool  shown  in  Table  3  are  contained  in  Appendix  B  through  Appendix  E.  Explanations 
of  the  criteria  used  in  the  surveys  are  contained  in  Appendix  A.  Both  the  structure  of  the  surveys  and  the 
selection  of  the  tools  reviewed  follow  from  our  consideration  of  the  USCG  acquisition  process  and  the  state- 
of-the-art  in  HSI  theory  and  tools. 
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Table  3.  HSI  tools  coverage.  (Key  to  abbreviations  at  end  of  table.) 


Tool  and  Category 

HSI  Domains 

HFE 

Training 

Safety 

Survivability 

Habitability 

Manpower 

Personnel 

3D  Static  Strength  Prediction  Program  (3DSSPP) 

C,D 

C,D 

C,D 

C,D 

2,4 

2,4 

2,4 

2,4 

Abstraction-Decomposition  Hierarchy  (MB) 

C,D 

C,D 

C,D 

C,D 

1,4 

1,4 

1,4 

1 

Adaptive  Control  of  Thought  -  Rational  (ACT-R)  (SB) 

C 

C 

C,1 ,2 

1,3 

1,3 

Advisor  3.5*  (SR) 

C,D,E 

2 

Applied  Cognitive  Task  Analysis  (ACTA)  (MB) 

C,D 

C,D 

C 

1,3 

1,3 

1 

Anthropometric  Accommodation  in  Aircraft  Cockpits  (--) 

C,D 

C,D 

C,D 

2,4 

2,4 

2,4 

Autonomous  Vehicle  Operator  Span  of  Control  Evaluation  Tool 

C 

C 

C 

(AVOSCET)  (SB) 

1,3 

1,3 

1,3 

C3TRACE  (SR) 

C.1,2 

Cognitive  Function  Analysis*  (CFA)  (MR) 

B,C,D 

B,C,D 

1,3,4 

1,2,4 

Cognitive  Reliability  and  Error  Analysis  Method  (CREAM)  (TR) 

C 

1 

C 

1 

Concept  Mapping*  (SB) 

A,B,C 

C 

C 

c 

C 

C 

C 

1,3 

1,2 

1 

1 

1 

1,2 

1,2 

Critical  Decision  Method  (MR) 

C 

C 

1,3,4 

1,3,4 

Designer’s  Situation  Awareness  Tool  (DeSAT)  (SR) 

C,D 

C,D 

1,3,4 

1,3,4 

Energy  Expenditure  Prediction  Program  (EEPP) 

C,D 

C,D 

C,D 

2,4 

2,4 

2,4 

Ergolntelligence  Upper  Extremity  Assessment  (UEA) 

C,D 

C,D 

C,D 

2,4 

2,4 

2,4 

Ergolntelligence  Manual  Materials  Handling  (MMH) 

C,D 

C,D 

2,4 

2,4 

Goals,  Operators,  Methods  and  Selection  Rules  (GOMS)  (MR) 

C 

C 

1,4 

1 
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Table  3.  HSI  tools  coverage  (continued). 


Tool  and  Category 

HSI  Domains 

HFE 

Training 

Safety 

Survivability 

Habitability 

Manpower 

Personnel 

Human  Factors  Workbench  (SB) 

C 

C 

C 

1,3,4 

1,3,4 

1,3,4 

IMPRINT  Pro/M icroSAI NT*  (SB) 

C,D 

C,D 

C,D 

C,D 

1,3,4 

2,4 

1,2,4 

2,4 

iGEN  (SB) 

C,D 

C,D 

C,D 

C,D 

1,3,4 

2,4 

1,2,4 

2,4 

Informal  System  Evaluation  Methods  (MB) 

B,C,D 

B,C,D 

B,C,D 

B,C,D 

B,C,D 

B,C,D 

B,C,D 

1,2, 3, 4 

1,2, 3, 4 

1,2, 3, 4 

1,2, 3, 4 

1,2, 3, 4 

1,2, 3, 4 

1,2, 3, 4 

Information  and  Functional  Flow  Analysis*  (IFFA)  (MB) 

B,C,D 

B,C,D 

C 

1 

1 

1,2 

Jack*  (SB) 

C,D 

C,D 

C,D 

1,4 

1,4 

1,4 

Job  Assessment  Software  System*  (JASS)  (SB) 

C,D 

C,D 

D 

D 

1,2 

1,2 

1,2 

1,2 

Liberty  Mutual  Tables  (--) 

C,D,2,4 

C,D,2,4 

LOCATE  (SR) 

C,D 

1,4 

Man-Machine  Integration  Design  and  Analysis  System  (MIDAS) 

C,D 

C,D,F 

(SR) 

1,3,4 

2,4 

Manpower  Analysis  and  Prediction  System  (MAPS)  (SR) 

C,D 

2 

C,D 

2 

Navy  Manpower  Requirements  System  (NMRS)  (SR) 

C,D 

2 

C,D 

2 

National  Institute  for  Occupational  Safety  and  Health  (NIOSH) 

C,D 

Lift  Equation 

2,4 

Risk  Management  Toolkit  (MB) 

B,C 

B,C 

B,C 

B,C 

B,C 

B,C 

B,C 

1 

1 

1,2,3 

1,2,3 

1,2,3 

1,2 

1,2 

Rapid  Upper  Limb  Assessment  (RULA) 

C,D 

C,D 

C,D 

2,4 

2,4 

2,4 

Ship-System  Human  Systems  Integration  for  Affordability  and 

B,C 

B,C 

B,C 

B,C 

B,C 

Performance  Engineering  (Ship-SHAPE)  Tool  Set  (SB) 

1,3,4 

1,2 

1 

1,2 

1,2 

Technique  for  Human  Error  Rate  Prediction*  (THERP)  (TR) 

D,E,F 

1 

D,E,F 

1 

*  Tools  included  in  the  “starter  kit”  (see  Sec.  3.1). 
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Key  to  codes  shown  in  Table  3. 


Tool  Type  and 
Coverage  Codes 

MB  =  Methodology 

Broad  HSI  Domain  Coverage 


Coast  Guard  Acquisition 
Phases 

A  -  Project  Identification 


B  -  Project  Initiation 

C  -  Concept  and  Technology 
Development 

D  -  Capability  Development  and 
Demonstration 

E  -  Production  and  Deployment 
F  -  Operations  and  Support 


Tool  Categories 

1  -  Simulation  &  Analysis  Tools  for 

CONOPS  and  Requirements 
Development 

2  -  Manning  and  Personnel 

Qualifications 

3  -  Workload  and  Situation 

Assessment  (SA) 

4  -  Workstation  and  Cockpit  Design 


MR  =  Methodology 

Restricted  HSI  Domain  Coverage 
SB  =  Software 

Broad  HSI  Domain  Coverage 
SR  =  Software 

Restricted  HSI  Domain  Coverage 
TB  =  Analysis  Technique 

Broad  HSI  Domain  Coverage 
TR  =  Analysis  Technique 

Restricted  HSI  Domain  Coverage 


3.1  A  Recommended  HSI  Tools  “Starter  Kit”  for  the  USCG 

Because  the  USCG’s  HSI  Technical  Authority  is  a  fledgling  program,  it  would  be  useful  to  consider  what 
tools  might  be  particularly  helpful  as  CG-1B3  establishes  its  acquisition  processes.  Given  the  tools  shown 
in  Table  3,  what  would  a  starter  kit  of  tools  for  HSI  in  acquisition  consist  of?  One  way  to  answer  this 
question  is  to  create  a  matrix  of  HSI  development  activities  against  acquisition  phases.  Useful  tools  can 
then  be  placed  in  appropriate  cells  of  the  matrix.  One  goal  would  be  to  minimize  the  number  of  tools 
needed,  while  maximizing  the  number  of  HSI  domains  that  the  tools  allow  analysts  to  address.  Using  these 
criteria,  we  have  formulated  a  minimum  HSI  tool  set  shown  in  Table  4. 

Table  4  presents  HSI  development  activities  down  the  left  and  the  CG  acquisition  phases  across  the  top. 

The  blue  shading  shows  the  acquisition  phases  in  which  the  different  HSI  development  activities  occur. 
Tools  from  the  starter  set  which  support  a  specific  development  activity  in  a  specific  acquisition  phase  are 
shown  in  the  cells  of  the  table.  The  first  thing  to  note  in  Table  4  is  that  most  of  the  tools  in  the  starter  kit  fall 
into  the  middle  two  phases  of  acquisition.  The  second  noteworthy  item  in  Table  4  is  that  the  kit  consists  of 
both  software  tools  and  methodologies.  The  third  item  of  note  is  that  the  starter  kit  tools  cover  almost  all  of 
the  HSI  development  activities  in  each  phase:  for  those  few  exceptions,  analysts  will  have  to  fall  back  on 
good  engineering  practice  until  acceptable  tools  are  developed  to  support  those  activities. 

The  selected  tools  allow  analysis  across  a  range  of  HSI  domains,  thereby  demonstrating  a  general  utility  for 
development  and  acquisition.  For  example,  the  concept  mapping  tool  can  be  used  to  carry  out  analysis  and 
represent  both  findings  and  issues  in  all  of  the  HSI  domains.  Likewise,  IMPRINT,  IFFA,  Jack,  and  JASS 
can  each  be  used  to  conduct  analyses  related  to  three  or  four  of  the  seven  HSI  domains.  Two  special- 
purpose  tools,  THERP  and  Advisor  3.5,  are  useful  in  rounding  out  coverage  of  the  above  tools  by  adding 
narrowly  defined,  but  highly  specific,  capabilities.  THERP  allows  analysts  to  identify  likely  errors  caused 
by  proposed  system  designs  and  estimate  system  failure  probabilities  associated  with  those  errors,  thereby 
allowing  enhanced  analysis  in  the  HFE  and  safety  domains.  Advisor  3.5  is  a  software  product  focused 
exclusively  on  issues  of  training.  This  tool  allows  analysts  to  estimate  the  time  commitments  and  costs  of 
specific  training  strategies  for  evolving  designs.  The  starter  tool  kit  provided  here  should  serve  HSI  analysts 
well  across  many  acquisition  phases  and  most  development  activities. 
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Most  of  the  tools  in  this  survey  address  multiple  development  activities.  The  collection  of  tools  in  the 
starter  kit  address  all  but  one  of  the  “standard”  HSI  development  activities  identified  earlier.  Furthermore, 
since  five  of  the  eight  tools  selected  for  the  starter  tool  kit  are  broad  in  scope,  they  should  allow  HSI 
analysts  to  address  a  wide  range  of  issues  likely  to  arise  in  system  development.  Thus,  the  presence  of  these 
tools,  as  well  as  their  breadth  of  coverage,  provides  a  pathway  by  which  HSI  considerations  can  be 
integrated  with  other  design  and  development  disciplines. 

The  manner  in  which  this  can  be  accomplished  is  through  (1)  an  integration  of  HSI  considerations  with 
evolving  system  requirements;  (2)  carrying  these  integrated  requirements  into  the  conceptual  system 
modeling  stages  of  design;  (3)  basing  initial  specifications  of  task  structure,  interfaces,  and  team  processes 
on  the  findings  of  HSI  analyses  and  doing  so  in  a  way  that  links  the  specifications  explicitly  to  the 
requirements  developed  for  the  overall  system;  (4)  providing  a  means  of  evaluating  the  evolving  system 
with  respect  to  the  effectiveness  of  the  system  and  indexed  back  to  the  originating  requirements;  and  (5) 
providing  a  means  to  ensure  that  performance  rises  to  the  level  required  by  the  operating  domain.  Tools  that 
did  not  contribute  to  the  accomplishment  of  one  or  more  of  these  objectives  were  not  included  in  the  starter 
kit. 


Table  4.  “Starter  kit”  of  HSI  tools. 


Development 

Activity 

Coast  Guard  Acquisition  Phase 

Project 

Identification 

Project 

Initiation 

Concept  & 
Technology 
Development 

Capability 
Development  & 
Demonstration 

Production 

& 

Deployment 

Operations 
&  Support 

Concept  Definition 

Concept 

mapping 

Concept 

mapping, 

CFA 

Concept 
mapping,  CFA 

Requirements  Analysis 

Concept 
mapping, 
CFA,  IFFA 

Concept 
mapping,  CFA, 
IFFA 

Function  Analysis  & 
Allocation 

CFA,  IFFA 
IMPRINT 

CFA,  IFFA, 
IMPRINT 

Task  Design 

Jack,  CFA, 
IMPRINT 

Jack, 

IMPRINT 

Interface/Team 

Development 

CFA 

CFA 

Performance/Workload/ 
Training  Estimation 

IMPRINT 

IMPRINT 

Requirements  Review 

Personnel  Selection 

JASS 

JASS 

Training  Development 

Advisor  3.5 

Advisor  3.5 

Advisor  3.5 

Performance 

Assurance  (T&E) 

THERP 

THERP 

THERP 

Note:  Blue  shading  in  body  of  table  indicates  HSI  development  activity  coverage  over  USCG  acquisition 

phases.  Colored  cells  with  no  tool  designation  indicate  a  lack  of  available  tools  to  support  the 
development  activities  in  these  phases. 
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3.2  A  Notional  Example  of  How  the  HSI  Tools  Starter  Kit  Could  Be  Applied  to 
Support  an  Acquisition 

We  now  present  an  example  of  how  the  HSI  tools  starter  kit  might  be  used  as  part  of  a  fictional  USCG 
system  development  project  in  order  to  illustrate  the  integration  methodology  outlined  above.  Assume  that 
the  following  need  has  been  identified  in  the  Project  Initiation  phase  of  an  acquisition. 

•  The  system  will  allow  C2  of  diverse  USCG  resources  operating  in  a  range  of  environments; 

•  The  C2  system  will  provide  dynamic  tracking  of  USCG  and  port  partner  asset  positions  and  status; 

•  The  C2  system  will  rely  on  a  database  that  is  populated  automatically  by  data  from  remote  sensors  and  manually  by  C2 
system  operators; 

•  The  C2  system  will  operate  at  tactical  (i.e.,  data  fusion  and  interpretation)  and  operational  (i.e.,  evaluation,  based  on  tactical 
results,  of  progress  toward  the  operational  plan)  levels,  and 

•  The  C2  system  will  be  responsible  for  producing  plans,  assessments,  and  recommendations  that  can  inform  command 
decision  makers  concerned  with  tactical  and  strategic  operations. 

Given  these  high-level  statements  of  need,  the  first  concern  of  the  HSI  analyst  would  be  to  understand  the 
purposes  and  constraints  associated  with  the  use  of  the  system.  The  analyst  probably  would  begin  by 
carrying  out  a  Cognitive  Systems  Engineering  (CSE)  analysis.  The  CSE  analysis  might  focus  on  several 
categories  of  information  that  can  be  used  to  document  the  work  domain,  the  work  itself,  and  the 
opportunities  and  constraints  that  will  be  important  in  carrying  out  the  work.  Specific  categories  in  the 
analysis  might  include  information  about  the  work  domain,  control  tasks,  strategies,  organizational  entities, 
socio-technical  factors,  and  worker  competencies  and  limitations.  Analysts  also  would  focus  on 
relationships  among  specific  items  of  information  in  each  of  these  categories.  The  concept  mapping 
software  tool  would  be  used  to  capture  and  archive  this  information,  as  shown  in  the  example  in  Figure  4. 


Acquisition  Directorate 

Research  &  Development  Center 


18 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


Figure  4.  A  concept  mapping  template  for  capturing  CSE  data  in  the  C2  example. 


As  the  information  in  the  concept  maps  is  successively  refined  through  increasing  levels  of  analysis,  it  will 
be  possible  for  the  analyst  to  identify  requirements  in  each  of  the  HSI  domains.  These  HSI  requirements 
can  then  be  integrated  with  those  from  other  engineering  disciplines  to  form  the  systems  engineering 
requirements  for  the  C2  system.  One  way  to  accomplish  this  is  to  follow  the  integration  methodology 
summarized  in  Figure  5.  This  methodology  maps  information  gathered  during  CSE  analysis  to  the  various 
types  of  system  engineering  analyses  conducted  early  in  system  development.  Note  that  the  HSI/CSE  data 
influence  almost  all  the  systems  engineering  analyses. 
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Figure  5.  A  methodology  to  integrate  CSE  analysis  results  with  other  systems  engineering  analyses. 

As  the  initial  CSE  analysis  proceeds  into  the  C&TD  acquisition  phase,  information  about  required  system 
operator  skills  and  aptitudes  will  begin  to  emerge.  The  JASS  tool  will  facilitate  the  capture  and  further 
definition  of  this  information.  An  example  of  a  JASS  information  capture  screen  is  shown  in  Figure  6. 
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Figure  6.  Information  capture  in  JASS  allows  analysts  to  specify  the  range  of  skills  needed  for  critical  tasks 
in  a  new  system. 


The  JASS  tool  is  based  on  accepted  descriptions  of  knowledge  categories,  skills,  and  aptitudes.  Use  of  this 
tool,  therefore,  would  allow  HSI  analysts  to  translate  information  from  the  concept  maps  into  a  form  that 
can  be  used  by  the  MPT  community  to  specify  required  attributes  for  operators  of  the  C2  system.  The 
output  of  the  JASS  tool  would  be  a  specification  of  the  knowledge,  skills,  and  aptitudes  of  system  operators 
that  can  be  used  for  personnel  selection.  This  information  also  will  be  invaluable  as  input  into  the 
development  and  implementation  of  a  training  plan.  As  this  personnel  and  aptitude  information  is  being 
specified  for  input  into  JASS,  analysts  also  could  develop  IMPRINT  models  to  study  the  broad  effects  of 
staff  sizes  and  configurations  on  high-level  system  effectiveness. 


Several  other  tools  also  will  be  useful  for  the  C&TD  phase.  For  example,  CFA  can  be  used  to  organize 
information  from  the  concept  maps  into  a  form  that  will  allow  analysts  to  identify  functions,  develop  logical 
functional  groupings,  and  formally  define  the  functions  that  operators  and  other  system  users  will  be 
expected  to  perform.  The  CFA  can  be  supplemented  with  an  IFFA  to  allow  specification  of  the  information 
requirements  for  each  function,  information  interdependencies  between  functions,  and  how  the  functions  “fit 
together”  to  define  the  workflow  of  the  system. 


This  is  a  crucial,  and  often-overlooked,  aspect  of  system  design.  In  the  absence  of  a  designed,  coherent 
workflow,  users  will  develop  a  workflow  that  they  can  use  to  accomplish  their  goals  and  required  tasks.  If 
workflow  organization  is  not  carefully  considered  during  system  design,  users  often  will  develop 
idiosyncratic  methods  that  decrease  system  effectiveness  and  can  introduce  safety  issues. 
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Use  of  CFA  and  IFFA  by  HSI  analysts  can  prevent  many  of  these  issues  from  arising  later,  after  the  system 
has  been  fielded.  Of  course,  when  these  types  of  system  models  have  been  defined  using  these 
methodological  tools,  they  should  then  be  used  to  estimate  system  effectiveness  and  support  trade  studies  so 
that  design  commitments  can  be  narrowed  to  those  most  likely  to  lead  to  “optimal”  design  outcomes. 
IMPRINT  will  be  particularly  useful  for  this  purpose,  and  should  be  a  tool  of  choice  at  this  point  in  the 
acquisition  cycle. 

Physical  task  design  also  becomes  prominent  during  the  C&TD  phase.  One  of  the  best  tools  available  to 
HSI  analysts  for  addressing  the  issues  in  this  area  is  the  Jack  bio-mechanical  modeling  environment.  Jack  is 
useful  for  analysis  and  design  of  the  physical  workspace.  It  can  be  used  to  conduct  link  and  reach  analyses, 
strength  assessments,  and  anthropometric  analysis,  as  exemplified  by  Figure  7.  Jack  will  allow  HSI  analysts 
to  conduct  trade  studies  and  address  areas  of  physical  fit,  fatigue,  and  comfort,  and  safety  issues  associated 
with  the  evolving  system  design. 


Figure  7.  Jack  allows  bio-mechanical  modeling  for  workspace  task  design. 


IMPRINT  also  allows  HSI  analysts  to  conduct  design  and  trade  studies  in  the  task  design  arena.  Its  focus, 
however,  is  on  areas  of  human  performance.  Figure  8  shows  how  HSI  analysts  can  develop  specific  task 
and  workflow  structures.  These  then  can  be  executed  to  assess  the  resulting  effects  on  system  performance, 
thereby  enabling  trade  studies  of  potential  work  alternatives.  Analysts  can  also  conduct  analyses  of  the 
workload  associated  with  alternative  task  structures  and  the  performance  changes  associated  with  these 
workload  variations.  Effects  on  performance  and  time  to  proficiency  of  various  training  alternatives  also 
can  be  evaluated. 
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Figure  8.  IMPRINT  supports  human  performance  trade  studies  for  task  design  and  workload  analysis. 


The  outputs  of  the  Jack  and  IMPRINT  tools  can  serve  as  inputs  to  Advisor  3.5,  enabling  HSI  analysts  to 
develop  and  manage  specific  training  designs.  Using  these  outputs  to  specify  particular  training  designs,  the 
Advisor  3.5  tool  (Figure  9)  will  allow  training  designers  to  determine  the  most  cost-effective  ways  to  deliver 
training  by  estimating  the  effectiveness  and  cost  of  alternative  methods,  estimating  the  time  required  to 
develop  materials  associated  with  different  training  alternatives,  and  computing  a  return  on  investment  for 
alternative  methods. 


Figure  9.  Advisor  3.5  allows  analysts  to  determine  timelines,  levels  of  effort,  and  cost  associated  with 
specific  training  alternatives. 
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Another  tool  that  will  be  useful  during  the  CD&D  phase  will  be  THERP.  As  the  system  design  evolves,  it 
will  become  possible  to  identify  likely  errors  that  operators  might  make  during  operations.  Using  THERP, 
HSI  analysts  can  categorize  the  errors,  estimate  their  probabilities  of  occurrence,  and  predict  the  likely 
consequences  of  each  error  on  system  effectiveness,  safety,  and  other  aspects  of  performance.  System 
designers  can  use  this  information  to  identify  design  alternatives  that  should  be  dropped  from  consideration 
due  to  these  system  effectiveness,  safety,  training,  or  other  issues.  A  highly  simplified  example  of  a  THERP 
fault  tree  is  shown  in  Figure  10. 

The  information  from  all  of  these  tools  accomplishes  several  goals.  They  allow  analysis  of  specific  aspects 
of  an  evolving  design  from  an  HSI  point  of  view.  They  do  so  in  ways  that  ensure  the  findings  and  analysis 
results  can  be  integrated  with  other  engineering  information  needed  to  arrive  at  an  effective  system  design. 
They  allow  HSI  analysts  to  conduct  critical  trade  studies,  needed  by  the  systems  engineering  community, 
that  will  inform  the  evolving  design.  That  is,  they  help  “point  the  design  team  in  the  right  direction”  with 
respect  to  design  commitments.  They  help  to  identify  likely  errors  and  estimate  the  impact  of  each  type  of 
error  on  system  safety  and  effectiveness.  They  help  HSI  analysts  to  define  and  integrate  task  and  workflow 
design  specifications  into  the  overall  system  definition  and  design  process.  Therefore,  they  provide 
invaluable  assistance  in  enabling  HSI  within  the  overall  engineering  of  new  systems. 


3.3  Human  Performance  Modeling:  Specialized  HSI  Tools 

As  the  USCG  acquires  more  complex  and  sophisticated  systems,  more  sophisticated  HSI  methods  and  tools 
are  needed  to  ensure  that  the  people  in  those  systems  support  and  enable  successful  mission  performance  by 
the  system.  Human  performance  models  (HPMs)  are  an  example  of  a  sophisticated  tool  that  helps  HSI 
practitioners  better  understand  human  performance  issues  in  complex  systems  and  address  those  issues  with 
effective  HSI  solutions.  This  section  provides  a  brief  overview  of  human  performance  modeling  and 
introduces  a  notional  example  (presented  in  Appendix  F)  of  how  the  technology  could  be  applied  to  help  the 
USCG  evaluate  the  addition  of  Unmanned  Aerial  Systems  (UAS)  to  the  National  Security  Cutter  (NSC). 
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As  noted  in  the  discussion  of  HSI,  understanding  the  work  requirements  of  people  in  a  system  is 
fundamental  to  the  HSI  process.  This  is  challenging  because  work  requirements  in  complex  systems  are 
dynamic,  driven  by  the  moment-to-moment  demands  of  the  mission  environment  and  the  hardware  and 
software  components  of  the  system.  Human  performance  modeling  provides  a  computational  means  for 
representing  these  dynamics  and  gaining  insights  into  human  performance  issues  such  as  numbers  and  types 
of  personnel  required,  levels  of  performance  proficiency  required  for  mission  success,  and  difficult  and 
mission-critical  performances  that  require  special  attention  in  terms  of  HFE  and  training  factors. 

There  are  a  number  of  human  performance  modeling  tools  available  to  HSI  practitioners  but  the  one  most 
widely  used  is  IMPRINT,  developed  by  the  Army  Research  Laboratory.  Figure  1 1  shows  a  portion  of  an 
IMPRINT  HPM.  IMPRINT  uses  task  network  modeling  to  represent  human  performance.  As  the  name 
implies,  task  networks  use  a  flowchart  type  format  to  specify  detailed  task  sequences,  decision  points,  and 
branches.  Task  networks  are  organized  under  higher-level  functions;  and  functions  are  organized  under 
goal  states.  Goals  states  are  a  concept  from  cognitive  psychology:  goals  provide  an  executive  function  that 
organizes  and  controls  behavior.  Goals  are  triggered  by  conditions  in  the  mission  environment  that  the 
performer  needs  to  maintain  within  certain  limits.  For  example,  a  navigation  goal  state  would  be  triggered 
in  a  pilot  when  he/she  determines  the  aircraft  is  off  course.  Once  the  goal  state  triggers,  lower-level 
functions  and  tasks  would  be  activated  to  bring  the  airplane  back  on  course  (restore  performance  within 
desired  limits). 

Modeling  environments  such  as  IMPRINT  generally  have  two  major  components.  The  first  is  a  model 
development  module.  In  IMPRINT,  this  is  a  graphical  interface  (Figure  1 1)  in  which  the  user  specifies 
goals,  functions,  and  tasks  and  arranges  their  organization  and  sequences.  The  user  also  enters  data  that 
control  how  the  goals,  functions,  and  tasks  execute.  These  data  include  trigger  (start)  criteria,  task 
performance  times  and  variances,  task  output  and  effects  data,  and  behavioral  process  algorithms  (e.g., 
visual  detection  and  identification,  information  processing,  etc.).  These  data  are  obtained  from  a  variety  of 
sources  that  include  the  human  factors  and  behavioral  science  literature  on  human  performance,  observation 
of  performance  in  operational  and  laboratory  environments,  and  interviews  with  subject  matter  experts.  The 
second  component  is  a  runtime  module.  This  module  actually  executes  the  model  (i.e.,  simulates  the 
sequence  of  tasks),  manages  time,  generates  and  collects  data,  and  manages  any  interfaces  to  other  models 
and  simulations. 
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Figure  11.  Sample  IMPRINT  model  screen. 

To  generate  the  dynamic  performance  effects  of  interest  to  HSI  practitioners,  events  are  incorporated  into  an 
HPM  that  represents  the  dynamic  demands  of  the  mission  environment  and  the  system  hardware  and 
software  components.  Environments  like  IMPRINT  provide  ways  for  modelers  to  create  these  events,  but 
the  most  preferred  method  is  to  integrate  the  HPM  with  simulations  of  the  system  hardware  and  software 
components  and  the  mission  environment.  Modeling  and  simulation  are  often  employed  in  the  development 
of  complex  systems  so  system  and  mission  environment  simulations  are  often  available.  When  the  HPM  is 
integrated  with  the  system  and  mission  environment  simulations,  the  result  is  a  complete  representations  of 
the  system  of  interest.  More  importantly,  these  “complete  system”  simulations  provide  more  realistic 
representation  of  the  performance  demands  that  stimulate  the  HPM.  Consequently,  performance  effects 
observed  in  the  output  of  the  HPM  are  more  realistic  and  predictive  of  what  can  be  expected  in  the  real 
system. 

The  ultimate  product  of  HPM,  and  indeed  all  models  and  simulations,  is  data  that  informs  decision  making. 
Typically,  multi-level  performance  measure  schemes  are  employed  that  systematically  assess  mission 
outcomes,  key  system  functions,  and  enabling  operating  components.  The  enabling  operating  components 
include  people  modeled  in  the  HPM.  The  multi-level  performance  measurement  schemes  help  decision¬ 
makers  understand  how  the  performance  of  human  and  other  system  components  propagate  through  the 
system  and,  ultimately,  affect  mission  outcomes.  By  systematically  manipulating  capabilities,  performance 
levels,  and  other  factors  and  observing  mission  impacts,  acquisition  decision-makers  can  converge  on  a  set 
of  objective  system  requirements  that  have  been  demonstrated  to  support  desired  levels  of  mission 
performance. 
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There  are  a  variety  of  factors  that  HSI  modeling  experts  can  manipulate  within  HPMs.  These  include  the 
numbers  of  personnel  that  perform  work  activities,  the  allocation  of  functions  and  tasks  across  those 
personnel,  and  the  levels  of  performance  generated  by  the  personnel.  The  levels-of-performance-generated- 
by-personnel  is  a  particularly  important  factor.  In  this  assessment,  performance  levels  (e.g.,  task 
performance  times,  performance  effects  such  as  accuracies  and  errors)  are  manipulated  systematically  to 
determine  the  levels  needed  to  enable  desired  levels  of  mission  performance.  Once  performance  level  data 
are  obtained,  the  data  are  reviewed  to  determine  whether  the  levels  are  reasonable  within  the  manpower  and 
personnel  concept  being  employed.  The  results  might  suggest  that  a  different  personnel  mix  (different 
numbers  and  types,  different  aptitudes  and  levels)  is  needed  to  generate  the  desired  levels  of  performance. 
Alternatively,  or  in  addition  to,  modifying  manpower  and-or  personnel  factors,  the  HSI  team  might  explore 
HFE  solutions  that  incorporate  tools  and  aids  that  enhance  human  performance  and-or  advanced  training 
concepts  that  generate  and  sustain  the  high  levels  of  performance  that  are  required.  HSI  modelers  would 
adjust  the  HPM  to  represent  these  different  approaches  to  HSI  solutions  and  test  them  within  the 
performance  assessment  framework  of  the  total  system  simulation.  When  combined  with  the  results  of 
other  HSI  domain  assessments  (e.g.,  safety,  habitability),  evolving  hardware  and  software  capabilities, 
system  cost  estimates,  and  risk  assessment,  a  clearer  understanding  of  the  best  HSI  solutions  and  their 
associated  system  requirements  will  emerge. 

A  final  point  to  be  made  about  human  performance  modeling  is  that,  like  other  forms  of  modeling  and 
simulation,  it  can  be  applied  throughout  the  acquisition  process.  Early  in  the  acquisition  process,  HPM  can 
be  used  to  support  development  of  CONOPS  for  new  systems.  Modeling  in  support  of  CONOPS  usually  is 
conducted  at  a  fairly  high  level  because  of  the  immaturity  of  the  system  concepts.  Similarly,  HPMs  used  to 
support  CONOPS  development  are  not  very  detailed  but  are  important  because  they  can  begin  to  define  and 
bound  the  requirements  for  human  performance  in  the  system.  Issues  such  as  key  personnel  roles  and 
functions  can  be  explored.  Also,  system  operational  doctrine  and  tactics  can  begin  to  be  considered.  This  is 
particularly  important  for  systems  that  are  incorporating  new  technologies  and  capabilities.  In  these 
systems,  an  important  question  posed  by  acquisition  decision-makers  is  “what  kinds  of  performance  gains 
can  I  expect  from  these  new  capabilities?”  The  answer  lies  not  in  the  technologies  themselves,  but  in  how 
they  are  applied.  The  application  of  technologies  in  a  new  system  is  “human  mediated,”  based  on  operating 
rules,  processes,  tactics,  etc.  By  incorporating  human  performance  into  the  early  modeling  and  simulation 
activities  used  to  develop  CONOPS,  the  engineering  team  has  the  opportunity  to  formulate  operational 
concepts  that  maximize  the  benefit  of  new  technologies  and  capabilities. 


As  the  acquisition  process  evolves  and  system  requirements  and  concepts  become  more  detailed,  the 
modeling  and  simulation  environments  become  more  detailed.  This  includes  the  human  performance 
modeling  environment.  The  evolutionary  nature  of  the  acquisition  process  allows  modelers  to  focus  the 
development  and  test  efforts  in  ways  that  minimize  cost  and  maximize  information  and  knowledge  gained. 
An  HPM  used  to  develop  CONOPS,  for  example,  might  represent  human  performance  simply,  i.e.,  to 
ensure  the  new  system  concept  provides  key  information  needed  to  support  decision-making.  Subsequent 
extensions  of  the  model  might  focus  on  representing  key  personnel  and  tasks  in  sufficient  detail  to  resolve 
the  numbers  and  types  of  personnel  and  the  appropriate  function  and  task  allocation  required  to  enable 
mission  success.  The  next  round  of  extensions  might  focus  on  very  detailed  representation  of  select, 
mission-critical  tasks  to  better  understand  operator-system  interface  requirements,  workload  and  situation 
awareness  issues  and  mitigation  strategies,  and-or  exceptional  proficiency  levels  that  training  must  generate 
and  maintain.  When  designed  properly,  a  high-level  HPM  developed  early  in  the  acquisition  process  should 
be  extensible  so  that  detail  can  be  added  selectively  as  the  system  concept  evolves.  This  makes  the 
application  of  human  performance  modeling  technology  much  more  cost  effective. 
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To  provide  the  reader  with  a  more  concrete  example  of  how  human  performance  modeling  is  applied, 
Appendix  F  presents  an  example  using  a  notional  study  of  potential  surveillance  mission  performance  gains 
obtained  by  adding  UAS  to  the  NSC.  The  example  addresses  the  overall,  notional  mission  context,  the  key 
system  elements  that  would  be  involved  in  the  study,  the  elements  of  human  performance  that  would  be 
modeled,  performance  measures  that  could  be  used,  and  possible  results.  Though  notional,  the  example 
does  highlight  some  key  points  about  the  use  of  human  performance  modeling,  including: 

1)  Human  performance  modeling  can  be  focused  and  scaled  to  address  key  human  performance  issues 
while  controlling  model  development  and  test  costs. 

2)  Smart  approaches  to  modeling  highlight  similarities  in  model  components  that  promote  reuse  and 
adaptation  of  model  elements  across  components  (e.g.,  elements  that  require  modeling  of  visual 
identification  of  vessels  can  be  reused  across  components  that  model  different  teams  -  such  as  a 
helicopter  crew  versus  a  UAS  sensor  operator  performing  visual  identification). 

3)  When  modeling  human  performance  in  detail,  attributes  of  a  situation  can  emerge  that  can  be  exploited 
in  tactics  and  process  development  to  improve  overall  mission  performance.  For  instance,  in  the  UAS 
example,  it  is  recognized  that  the  UAS  speed  and  sensor  range  combine  to  ensure  that  vessels  in  the 
operational  area  (OP AREA)  are  detected  multiple  times.  This  provides  multiple  opportunities  to  correct 
target  misidentifications  made  previously.  Well-defined  tactics  and  procedures  will  provide  a  means  of 
exploiting  this  opportunity. 

3.4  Survey  of  Specific  HSI  Tools 

With  the  exception  of  Appendix  F,  as  described  above,  the  remainder  of  this  report  focuses  on  the  HSI  tools 
survey,  including  an  introduction  to  the  criteria  used  to  evaluate  the  tools  (Appendix  A),  followed  by  the 
surveys  themselves  (Appendix  B  through  Appendix  E). 

We  begin  the  survey  discussion  with  an  explanation  of  the  criteria  used  in  each  tool  review  (Appendix  A). 
These  criteria  are  organized  into  four  sections  consisting  of  those  focused  on  (1)  content  considerations  of 
each  tool;  (2)  the  manner  in  which  tool  outputs  are  communicated  to  the  larger  system  development  process; 
(3)  facilities  provided  by  the  tools  for  helping  manage  risks  and  trade-offs  across  HSI  domains  and  between 
HSI  and  other  system  development  considerations;  and  (4)  the  manner  in  which  a  tool  allows  HSI  analysts 
to  relate  the  form  and  function  of  artifacts  to  the  requirements  of  the  artifact.  Each  survey  consists  of 
several  specific  items  in  each  focal  area  upon  which  the  tool  is  reviewed.  See  Appendix  A  for  detailed 
definitions  of  these  items. 

As  was  shown  in  Table  3,  we  surveyed  34  tools  that  addressed  one  or  more  domains  of  HSI.  The  tool 
surveys  are  presented  in  four  appendices  designed  to  organize  them  into  sets  addressing  the  major  HSI 
concerns  within  a  system  acquisition. 

Appendix  B  presents  simulation  and  analysis  tools  for  CONOPS  and  requirements  development,  including: 

•  Abstraction-Decomposition  Hierarchy  •  Concept  Mapping 

•  ACT-R  •  MIDAS 

•  Critical  Decision  Method  •  GOMS 

•  Cognitive  Function  Analysis  •  iGen 
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•  Information  and  Functional  Flow  Analysis  •  AVOSCET 

•  Applied  Cognitive  Task  Analysis 

Appendix  C  presents  tools  for  determining  manning  and  personnel  qualifications,  including: 

•  Advisor  3.5  •  Navy  Manpower  Requirements  System 

•  Job  Assessment  Software  System  •  C3TRACE 

•  Manpower  Analysis  and  Prediction  System 

Tools  under  Appendix  D  are  for  workload  and  situation  assessment,  and  include: 

•  CART/IMPRINT  Pro 

•  Designer’s  Situation  Awareness  Tool 

•  Cognitive  Reliability  and  Error  Analysis  Method 

•  THERP 


Workstation  and  cockpit  design  tools  are  in  Appendix  E,  including: 


•  3D  Static  Strength  Prediction  Program 

•  Energy  Expenditure  Prediction  Program 

•  LOCATE 

•  Liberty  Mutual  manual  material  handling 
tables 

•  NIOSH  lift  equation 

•  informal  system  evaluation  methods 

•  Human  Factors  Workbench 


•  Rapid  Upper  Limb  Assessment 

•  Ship-SHAPE  Tool  Set 

•  Risk  Management  Toolkit 

•  Jack 

•  Ergolntelligence  Upper  Extremity  Assessment 

•  Ergolntelligence  Manual  Material  Handling 

•  Anthropometric  Accommodation  in  Aircraft 
Cockpits 


In  order  to  use  this  survey  to  identify  a  tool  for  use  during  acquisition  and  development,  first  consult  Table  3 
in  order  to  identify  candidates  that  might  satisfy  your  analysis  needs  for  the  acquisition  phases  and  tool 
categories  of  interest.  When  candidates  have  been  identified  from  Table  3,  analysts  can  locate  them  in  one 
of  the  four  appendices  shown  in  the  “tool  categories”  designation.  Reviewing  the  surveys  in  the  appropriate 
appendix  will  indicate  what  system  development  activities  each  tool  addresses,  along  with  other  information 
about  the  nature  of  the  tool.  The  “best”  tool  should  then  be  selected  based  on  these  survey  criteria. 
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APPENDIX  A.  SURVEY  CRITERIA  FOR  HSI  TOOLS 


The  tools  included  in  the  survey  were  evaluated  along  four  broad  criteria  that,  we  feel,  must  be  addressed  in 
order  for  HSI  considerations  to  be  satisfactorily  integrated  into  the  Coast  Guard  system  acquisition  process. 

Criterion  1:  Content.  The  first  criterion  is  that  of  the  content  produced  by  the  tool.  In  order  to  be  useful 
along  this  dimension,  a  tool  should  produce  information  bearing  on  design,  rather  than  producing 
information  in  the  service  of  theory  testing.  The  output  also  should  be  articulated  at  a  level  of  fidelity  that 
matches  the  mission-oriented  considerations  of  system  design.  The  content  produced  by  the  tool  should  be 
actionable,  that  is,  it  should  support  the  derivation  of  design  commitments.  It  also  should  produce  content 
that  can  be  used  to  support  trade-offs  that  will  have  to  be  made  during  the  course  of  the  development  effort. 
Finally,  good  tools  should  support  designers  in  understanding  how  adaptive  behavior  at  one  level  influences 
of  constrains  behavior  at  other  levels  throughout  the  operating  range  of  the  system. 

The  content  of  each  tool  was  assessed  with  respect  to  a  number  of  measures. 


a.  Theoretical  assumptions.  These  are  the  assumptions  about  the  structure  and  organization  of  basic 
processes  that  both  shapes  and  constrains  the  types  of  analyses  the  tool  can  be  used  to  conduct,  the 
terminology  and  conventions  a  tool  adopts  and  the  mechanisms  by  which  analyses  are  carried  out. 
For  example,  a  software  tool  might  be  described  as  “a  theory  of  cognitive  functioning  embodied  as  a 
computer  model.”  The  assumptions  made  by  the  theory  about,  say,  memory  dynamics  would 
influence  any  simulations  of  operator  performance  within  the  context  of  specific  system  designs. 

b.  HSI  domains  addressed.  This  is  an  enumeration  of  the  domains  that  each  tool  addresses,  either 
directly  or  indirectly.  All  tools  address  at  least  one  domain  directly,  that  is,  the  tool  is  designed  to 
permit  analyses  or  design  activities  in  the  target  domain  explicitly.  For  example,  a  human  reliability 
tool  would  directly  address  the  safety  domain  through  its  analysis  of  error  probabilities  and 
probabilities  of  system  failures  arising  from  the  errors.  Tools  also  can  address  domains  indirectly  by 
providing  analyses  or  information  that  can  be  used  to  make  design  decisions  related  to  those 
domains.  A  simulation  environment  that  produces  output  regarding  workload  under  different 
conditions  of  stress  indirectly  addresses  safety  by  providing  information  that  can  be  used  in  an  error 
and  reliability  analysis. 

c.  Questions  addressed.  Each  tool  addresses  certain  questions  with  respect  to  the  activities  involved  in 
system  design,  aspects  of  HSI  domains,  and  phases  of  Coast  Guard  acquisition.  We  attempted  to 
keep  all  three  of  these  areas  in  mind  as  we  reviewed  each  tool.  Broadly  speaking  these  questions 
tended  to  cluster  into  six  categories:  work  structure,  effectiveness,  performance,  optimization, 
management,  and  cost  and  return  on  investment. 

d.  Content  modeled.  The  specific  HSI  content,  in  each  HSI  domain,  addressed  by  the  tool. 

e.  Granularity.  Different  tools  carry  out  their  analyses  and  produce  their  outputs  at  differing  levels  of 
granularity.  Whereas  one  tool  might  allow  only  identification  of  broad  categories  of  processes  (e.g., 
engage  checklist),  another  tool  might  allow  detailed  description  of  the  process  (e.g.,  a  detailed  task 
network  of  checklist  use)  in  conjunction  with  completion  time  estimates  of  each  process  step. 
Clearly,  these  two  tools  will  support  different  design  activities,  and  at  different  levels  of  fidelity. 

The  tools  in  this  survey  ranged  across  many  of  levels  of  granularity.  We  assessed  the  granularity  of 
a  tool  as  either  low  of  high.  This  assessment  does  not  reflect  a  rigorous  or  precise  measure  of  tool 
granularity;  rather,  we  applied  subjective  judgments  to  each  tool  based  on  either  direct  experience  or 
the  description  provided  by  the  tool  vendor. 
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Criterion  2:  Communication.  A  closely  related  criterion  is  that  of  communication.  The  challenge  here  is  to 
communicate  information  to  the  other  developmental  disciplines  in  a  way  that  facilitates  their  use  of  the 
information  for  their  own  design  activities.  Although  there  are  several  “models”  of  interdisciplinary 
communication  between  the  HSI  and  system  development  communities,  the  one  we  favor  places  the 
responsibility  for  effective  communication  on  the  HSI  community.  Accordingly,  the  content  of  tool  output 
produced  by  the  HSI  community  should  be  stated  in  language,  and  use  conventions,  that  are  conducive  to 
easy  incorporation  by  other  developers.  When  HSI  tools  use  language  and  conventions  to  communicate 
effectively,  they  will  facilitate  integration  of  HSI  considerations  into  acquisition  activities  and  will  improve 
the  chances  that  HSI  practitioners  can  become  full,  contributing  members  of  interdisciplinary  development 
teams. 

The  communication  effectiveness  of  each  tool  was  surveyed  with  respect  to  several  indexes. 

a.  Form  of  output,  terminology,  and  taxonomic  conventions .  This  area  addresses  the  manner  in  which 
content  produced  by  a  tool  is  expressed.  In  most  cases,  the  tools  falling  into  HSI  domains  render 
their  output  as  text,  statistical  analysis  results,  graphical  representations  and  so  on.  These  renderings 
must  be  “translated”  into  other  forms  for  direct  use  by  the  systems  engineering  and  development 
functions.  Ideally,  HSI  analysis  output  would  be  rendered  in  a  manner  that  could  be  directly  input  to 
engineering  analysis.  Since  this  is  not  the  case  currently,  we  judge  the  “better”  HSI  tools  to  be  those 
that  approximate  renderings  appropriate  to  engineering  modeling  and  analysis  needs.  Terminology, 
definitions,  and  taxonomic  conventions  address  the  specific  language  used  in  the  output  of  tool 
content.  Once  again,  the  degree  of  approximation  to  engineering  analysis  needs  is  a  critical 
consideration.  Some  tools  speak  only  in  the  language  of  the  tool:  Cognitive  processes,  workload, 
situation  awareness,  and  so  on.  The  information  produced  by  these  tools  will  not  often  be  used  by 
the  “engineering  process”  because  engineers  are  not  sure  how  to  use  the  terminology,  definitions  and 
conventions  as  data  for  engineering  analysis  and  design.  Thus,  a  tool  that  produces  data  about 
human  visual  detection  performance  with  a  terminology  based  only  on  detection  probabilities, 
degrees  of  visual  angle,  foveal  attention,  and  iconic  persistence  is  not  likely  to  be  useful  to  a  process 
that  needs  this  information  presented  in  terms  of  system  effectiveness. 

b.  Methods  of  integration  with  system  development  processes.  Results  from  HSI  analyses  can  be 
integrated  in  several  ways. 

i.  Provide  data  that  can  be  used  by  engineering  analyses.  These  data  might  address  aspects  of 
human  performance,  capabilities,  or  limitations  that  will  be  important  in  designing  the  system 
and-or  performing  the  trade-offs  that  will  inevitably  be  required  during  system  development. 
Another  way  in  which  HSI  analyses  can  be  used  directly  by  engineering  analyses  is  by 
supporting  the  modeling  of  some  aspect  of  human  work  that  will  be  important  in 
development  of  the  system. 

ii.  Identify  important  parameters  or  other  factors  affecting  the  performance  of  a  system.  HSI 
analyses  might,  for  example,  identify  the  parameters  that  must  be  considered  in  designing  a 
system  for  single  operator  control  of  multiple  uninhabited  vehicles. 

iii.  Provide  design  guidance  regarding  some  aspect  of  the  overall  system  design.  Principles 
based  on  archival  data  might  be  used  to  provide  guidance  in  the  design  of  feedback  lags  in  a 
system,  for  example. 
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iv.  Provide  specifications  that  define  a  system  or  subsystem.  HSI  analyses  often  are  used  to 
develop  specifications  for  the  interface  in  a  system,  or  more  recently,  to  develop 
specifications  for  advanced  visualizations  to  be  used  in  a  system.  Specifications  regarding 
system  safety,  habitability  and  other  areas  also  are  commonly  developed  based  on  HSI 
analyses. 

v.  Provide  methods  for  evaluation  of  a  system  or  subsystem.  Several  of  the  tools  discussed 
below  are  designed  to  provide  ways  of  evaluating  aspects  of  systems  informally  and 
formally,  and  in  formative  and  engineering  phases  of  development.  To  the  extent  that  these 
tools  are  used  in  an  integrated  manner  across  the  course  of  system  development,  they  can 
help  to  keep  the  development  “on  course”  by  spotting  problems  and  resolving  uncertainties. 

c.  Language  and  interface  support.  This  area  addresses  the  computer  language,  if  any,  used  by  each 
tool  in  the  survey.  We  also  address  here  the  degree  and  type  of  interface  support  provided  by  the 
tool.  In  some  cases,  this  will  be  an  integrated  part  of  the  tool,  as  in  the  case  of  a  simulation 
environment  or  custom  computer-based  tool.  In  other  cases,  there  might  be  no  interface  support,  as 
in  the  case  of  a  methodology  or  analysis  technique.  A  third  possibility  is  that  a  tool  might  rely  on 
third-party  interfaces  such  as  MS  Office  applications  or  statistical  packages. 

Criterion  3:  Risk  and  Trade-offs.  The  third  criterion  used  in  our  tool  survey  was  that  of  risk  assessment  and 
trade-off  management.  An  effective  tool  should  allow  analysts  to  identify,  quantify,  and  assess  the  likely 
consequences  of  risks  associated  with  specific  issues  arising  across  HSI  domains.  The  most  useful  tools 
will  allow  analysts  to  identify  risks  associated  with  various  HSI  considerations,  quantify  their  likelihoods  of 
occurrence,  and  estimate/mitigate  their  potential  detrimental  consequences.  One  way  this  can  be  done  is  to 
provide  information  needed  to  avoid  developing  incomplete  or  inaccurate  requirements.  This  is,  in  fact,  a 
strength  of  many  of  the  tools  included  in  this  survey,  albeit  at  a  qualitative  level.  Some  of  the  tools  even  go 
beyond  this  by  supporting  analysts  in  developing  quantitative  estimates  of  errors  and  their  consequences. 

Risk  management  and  trade-off  support  was  assessed  according  to  the  following  factors. 

a.  Assumptions  regarding  risk.  This  item  addresses  whether  the  tool  makes  any  assumptions  about  the 
nature  of  risk  and,  if  so,  what  those  assumptions  are.  Most  tools  do  not  address  this  area.  Only  those 
tools  designed  to  be  used  as  human  reliability  instruments  explicitly  address  this  area.  A  few  tools 
that  address  other  HSI  domains  will  also  address  risk,  perhaps  indirectly.  Examples  of  such  tools 
would  be  those  concerned  with  workload  or  those  supporting  simulation  of  human  performance. 

b.  Error/Risk  definitions  and  taxonomy.  If  risk  is  addressed  by  a  tool,  this  category  will  address  the 
taxonomy  used  in  supporting  the  risk  assessments. 

c.  Risk  computation/mitigation  methodology.  For  tools  that  explicitly  address  risk,  this  category 
discusses  the  analysis  and  computation  methods  used  to  measure  the  risk,  assess  the  consequences  of 
the  risk,  and  develop  mitigation  strategies. 

Criterion  4:  Traceability.  The  fourth  criterion  used  in  the  survey,  traceability,  is  concerned  with  linking 
developed  artifacts  to  originating  requirements.  Tools  that  are  maximally  useful  should  provide  ways  of 
accomplishing  this  linkage.  Unfortunately,  this  criterion  is,  arguably,  the  most  difficult  one  for  HSI  tools  to 
satisfy.  Reasons  for  this  stem,  in  part,  from  inadequacies  related  to  the  other  criteria.  For  example,  if  a  tool 
does  not  support  production  of  the  type  of  content  needed  to  establish  the  linkages  that  constitute 
traceability,  then  the  traceability  criterion  will  not  be  met.  Fikewise,  if  a  tool  does  not  communicate  in  a 
manner  conducive  to  the  establishment  of  traceability,  then  this  criterion  will  not  be  met. 
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Three  areas  were  used  to  survey  the  traceability  provided  by  each  tool. 

a.  Output  categories  and  relationships  to  acquisition  requirements.  Output  categories  include  the 
categories  of  information  that  the  tool  produces.  These  can  include  data,  descriptions,  specifications, 
analysis  results,  evaluations,  requirements,  and  models.  These  can  be  related  to  acquisition 
requirements  by  contributing  directly  to  acquisition  control  documents  such  as  the  MNS,  PORD, 
ORD  and  so  on;  by  serving  as  criteria  in  milestone  decision-making  such  as  decisions  to  proceed  to 
subsequent  analysis  phases;  by  contributing  to  the  development  of  design  specifications;  and  by 
serving  as  input  to  program  reviews. 

b.  Development  steps  supported.  This  will  be  a  listing  of  the  design  and  development  activities  that  a 
tool  supports. 

c.  Method  of  mapping  system  requirements  to  HSI  work  requirements.  This  addresses  the  question  of 
whether  the  tool  provides  an  explicit  way  of  relating  system  requirements  to  HSI  requirements.  A 
critical  element  in  the  notion  of  traceability  is  whether  design  commitments  in  HSI  domains  can  be 
traced  back  to  related  system  requirements.  It  should  be  stated  that,  in  most  cases,  the  tools  surveyed 
here  do  not  provide  an  explicit  means  of  doing  that.  In  some  cases,  however,  a  tool  will  support  the 
development  of  a  method  to  carry  out  this  traceability. 

In  addition  to  the  four  criteria  outlined  above,  the  review  of  each  tool  concludes  with  a  section  on  the 
learning  curve  to  become  proficient  with  the  tool,  general  information  regarding  tool  support,  and  contact 
information. 

The  information  on  learning  curve  has  been  included  as  a  general  guide  to  the  estimated  time  that  would  be 
required  to  develop  sufficient  facility  with  the  tool  to  use  it  profitably  in  development  projects.  Three  levels 
are  used  to  estimate  learning  curves.  Tools  with  shallow  curves  are  assumed  to  be  “leamable”  within  1 
week.  The  content  and  assumptions  associated  with  these  tools  are  assumed  to  be  easy  to  understand. 

There  are  no  special  skills  needed  to  begin  using  the  tool  profitably  and  the  tool  is  assumed  to  require  little 
experience  to  apply  to  design  challenges.  Tools  with  moderate  learning  curves  are  estimated  to  require 
about  2  weeks  to  “master.”  These  tools  have  content  and  assumptions  that  require  users  to  engage  in  some 
amount  of  learning,  and  perhaps  some  background  research,  prior  to  being  able  to  apply  the  tool.  Some 
special  skills  might  be  required  of  users,  perhaps  some  acquisition  of  (minimal)  programming  skills. 
Background  experience  will  be  useful  in  learning  to  use  the  tool.  Some  minimal  training  might  be  needed. 
Tools  with  steep  learning  curves  will  require  more  than  2  weeks  to  proficiency.  Substantial  learning  will  be 
required  to  master  the  content  and  assumptions  of  the  tool.  Special  skills  are  required  to  use  to  the  tool 
competently.  Users  will  be  required  to  either  receive  substantial  training  or  participate  in  substantial 
practice  before  being  able  to  use  the  tool  to  solve  design  problems. 

Readers  will  note  that  some  HSI  categories  are  only  sparsely  represented  in  this  survey.  In  particular, 
survivability  and  habitability  have  few  tools  representing  them.  These  domains  fall  outside  the  scope  of  the 
survey  requested  by  the  CG  R&D  Center  because  other  CG  organizations  already  deal  with  these  issues. 

The  wealth  of  tools  that  address  these  categories,  therefore,  were  not  included  here. 
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B.l  Abstraction-Decomposition  Hierarchy 

Description 

Hierarchical  description  of  system  operations  that  relates  system  purpose  to  physical  form  through  five 
levels  of  decomposition. 

Tool/Method  Content 

Theoretical  Assumptions: 

•  Environment  is  a  primary  determining  factor  in  system  behavior/effectiveness 

•  Systems  are  teleological  (goal-seeking)  artifacts 

•  Each  level  of  abstraction  in  a  system  must  be  related  to  other  levels  for  the  system  to  achieve 
resilience  in  the  face  of  environmental  complexity 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

•  Safety 

•  Personnel 


Questions  Addressed: 

•  System  behavior  under  environmental  variation 

•  Control  tasks 

•  Adaptive  strategies 

Content  Modeled: 

•  Functional  purpose 

•  Abstract  function 

•  Generalized  function 

•  Physical  function 

•  Physical  form 

Model  Granularity: 

Extends  from  high  level  functional  purpose  to  low  level  physical  form. 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

•  Abstraction  -  decomposition  hierarchy 

•  Control  tasks 

•  Strategies  for  carrying  out  control  tasks 


Methods  used  to  Integrate  with  SE  and  Other  Environments: 

No  explicit  points  of  integration.  Often  proposed  as  a  replacement  for  conventional  Systems  Engineering 
(SE)  process,  though  this  approach  is  not  feasible.  Best  integration  point  is  in  early  analysis  as  an  input  to 
system  requirements  analysis  and  modeling.  No  explicit  methods.  Technique  is  analytical  only.  Can  be 
integrated,  in  principle,  with  any  environment. 


Acquisition  Directorate 

Research  &  Development  Center 


B-2 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


Computer  Language  and  Interface  Support: 

None,  this  is  an  analytical  technique  only.  “Interface”  is  left  up  to  the  analyst. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Not  explicitly  part  of  this  tool. 

Error  Definitions/Taxonomy: 

Systems  are  under  development  to  include  error  analysis  in  the  general  scope  of  the  tool.  These  are  not 
likely  to  be  a  formal  tool. 

Risk  Computation/Mitigation  Methodology: 

None  at  this  time.  Erik  Hollnagel  has  developed  a  methodology  for  error  estimation.  Not  currently 
integrated. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Output  categories  include 

•  System  purpose 

•  Constraints 

•  General  laws  that  apply  to  system  operations 

•  Physical  functions  used  to  realize  system  purpose 

•  Physical  forms  (interfaces) 

Methodology  allows  designers  to  identify  environmental  factors  affecting  system  behavior  &  effectiveness, 
and  relate  these  to  human  operator  work  strategies  and  performance  competencies. 

Development  Steps  Supported: 

•  Cognitive  work  analysis 

•  System  requirements  analysis 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Allows  analysis  that  will  help  identify  requirements  in  each  HSI  domain. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

N/A 

Analysis  Utilities  and  Interface  Support: 

N/A 

Availability,  Cost,  and  Contact  Information: 

Vicente,  K.  (1999).  Cognitive  Work  Analysis.  Erlbaum:  Hillsdale,  NJ. 
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B.2  Adaptive  Control  of  Thought  -  Rational  (ACT-R) 

Description 

ACT-R  is  a  “hybrid”  cognitive  architecture  that  aspires  to  provide  an  integrated  account  of  many  aspects  of 
human  cognition.  It  is  a  successor  to  the  previous  ACT  production- system  theories,  with  emphasis  on 
activation-based  processing  as  the  mechanism  for  relating  a  production  system  to  a  declarative  memory. 

ACT-R  as  originally  developed  was  a  model  of  higher- level  cognition.  That  model  has  been  applied  to 
modeling  domains  such  as  Tower  of  Hanoi,  mathematical  problem  solving  in  the  classroom,  navigation  in  a 
computer  maze,  computer  programming,  human  memory,  learning,  and  other  tasks. 

ACT-R  is  a  cognitive  architecture:  a  theory  about  how  human  cognition  works.  On  the  exterior,  ACT-R 
looks  like  a  programming  language;  however,  its  constructs  reflect  assumptions  about  human  cognition. 
These  assumptions  are  based  on  numerous  facts  derived  from  psychology  experiments. 

Like  a  programming  language,  ACT-R  is  a  framework:  for  different  tasks  (e.g.,  planning  tasks,  memory  for 
text  or  for  list  of  words,  language  comprehension,  communication,  aircraft  controlling),  researchers  create 
models  (a.k.a.  programs)  that  are  written  in  ACT-R  and  that,  beside  incorporating  the  ACT-R’s  view  of 
cognition,  add  their  own  assumptions  about  the  particular  task.  These  assumptions  can  be  tested  by 
comparing  the  results  of  the  model  with  the  results  of  people  doing  the  same  tasks.  By  “results”  we  mean 
the  traditional  measures  of  cognitive  psychology: 

•  Time  to  perform  the  task, 

•  Accuracy  in  the  task,  and, 

•  Neurological  data  such  as  those  obtained  from  Functional  Magnetic  Resonance  Imagery  (fMRI). 

Tool/Method  Content 

Theoretical  Assumptions: 

In  general,  ACT-R  adheres  to  the  assumptions  inherent  in  the  ACT,  with  the  minor  exception  that  all 
processors,  including  the  motor  processors,  communicate  through  the  contents  of  working  memory,  and 
not  directly  from  cognition. 

ACT-R  assumes  that  there  are  two  types  of  knowledge — declarative  and  procedural — and  that  these  are 
architecturally  distinct.  Declarative  knowledge  is  represented  in  terms  of  chunks,  which  are  schema-like 
structures  consisting  of  an  is  a  pointer  specifying  their  category  and  some  number  of  additional  pointers 
encoding  their  contents.  Procedural  knowledge  is  represented  in  production  rules.  ACT-R’s  pattern 
matching  facility  allows  partial  matches  between  the  conditions  of  productions  and  chunks  in  declarative 
memory. 

Both  declarative  and  procedural  knowledge  exist  permanently  in  long-term  memory.  Working  memory  is 
the  portion  of  declarative  knowledge  that  is  currently  active.  Thus,  the  limitation  on  working  memory 
capacity  in  ACT-R  concerns  access  to  declarative  knowledge,  not  the  capacity  of  declarative  knowledge. 

HSI  Domains  Addressed: 

•  HFE 

•  Personnel 

•  Training 
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Questions  Addressed: 

ACT-R  has  been  used  in 

•  Human-computer  interaction  (HCI)  to  produce  user  models  that  can  assess  different  computer 
interfaces, 

•  Education  (cognitive  tutoring  systems)  to  “guess”  the  difficulties  that  students  may  have  and  provide 
focused  help, 

•  Computer-generated  forces  to  provide  cognitive  agents  that  inhabit  training  environments, 

•  Neuropsychology,  to  interpret  FMRI  data. 

Content  Modeled: 

Like  a  programming  language,  ACT-R  is  a  framework:  for  different  tasks  (e.g.,  Tower  of  Hanoi,  memory 
for  text  or  for  list  of  words,  language  comprehension,  communication,  aircraft  controlling),  researchers 
create  models.  These  models  reflect  the  modelers’  assumptions  about  the  task  within  the  ACT-R  view  of 
cognition.  The  model  might  then  be  run. 

Running  a  model  automatically  produces  a  step-by-step  simulation  of  human  behavior  which  specifies 
each  individual  cognitive  operation  (i.e.,  memory  encoding  and  retrieval,  visual  and  auditory  encoding, 
motor  programming  and  execution,  mental  imagery  manipulation).  Each  step  is  associated  with 
quantitative  predictions  of  latencies  and  accuracies.  The  model  can  be  tested  by  comparing  its  results  with 
the  data  collected  in  behavioral  experiments. 

Model  Granularity:  Very  high.  Processes  can  be  modeled  down  to  the  neurological  level. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Modules.  There  are  two  types  of  modules: 

•  perceptual-motor  modules,  which  interface  with  a  simulation  of  the  real  world.  The  most  well- 
developed  perceptual-motor  modules  in  ACT-R  are  the  visual  and  the  manual  modules. 

•  memory  modules.  There  are  two  kinds  of  memory  modules  in  ACT-R: 

o  declarative  memory,  consisting  of  facts  such  as  Washington,  D.C.  is  the  capital  of  United 
States,  France  is  a  country  in  Europe,  or  2+3=5,  and 
o  procedural  memory,  made  of  productions.  Productions  represent  knowledge  about  how  we 
do  things:  for  instance,  knowledge  about  how  to  type  the  letter  “Q”  on  a  keyboard,  about 
how  to  drive,  or  about  how  to  perform  addition. 

Buffers.  ACT-R  accesses  its  modules  (except  for  the  procedural-memory  module)  through  buffers.  For 
each  module,  a  dedicated  buffer  serves  as  the  interface  with  that  module.  The  contents  of  the  buffers  at  a 
given  moment  in  time  represents  the  state  of  ACT-R  at  that  moment. 

Pattern  Matcher.  The  pattern  matcher  searches  for  a  production  that  matches  the  current  state  of  the 
buffers.  Only  one  such  production  can  be  executed  at  a  given  moment.  That  production,  when  executed, 
can  modify  the  buffers  and  thus  change  the  state  of  the  system.  Thus,  in  ACT-R  cognition  unfolds  as  a 
succession  of  production  firings. 
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Most  existing  ACT-R  models  stand  alone;  all  of  the  action  is  cognitive,  while  perception  and  motor 
behavior  are  finessed.  However,  some  models  have  been  built  that  interact  with  an  external  world 
implemented  in  Macintosh  Common  Lisp  or  HyperCard™.  The  ACT-R  models  generally  only  interact 
with  custom  systems  developed  by  the  modeler,  rather  than  with  a  commercially-available  system. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

None 

Computer  Language  and  Interface  Support: 

Written  in  Common  Lisp  and  programmed  for  easy  extensibility. 

In  addition  to  the  fully  functional  and  portable  implementation  of  the  ACT-R  system,  a  number  of  tools 
are  available.  There  is  a  graphical  environment  for  the  development  of  ACT-R  models,  including  a 
structured  editor;  inspecting,  tracing,  and  debugging  tools;  and  built-in  tutoring  support  for  beginners.  A 
perceptual/motor  layer  extending  ACT-R’ s  theory  of  cognition  to  perception  and  action  is  also  available. 
This  system,  called  ACT-R/PM,  consists  of  a  number  of  modules  for  visual  and  auditory  perception, 
motor  action,  and  speech  production,  which  can  be  added  in  modular  fashion  to  the  basic  ACT-R  system. 
Both  ACT-R  and  ACT-R/PM  are  currently  available  for  the  Macintosh,  but  there  are  plans  to  port  them  to 
the  Windows  platform  or  to  some  platform-independent  format,  such  as  CLIM. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk:  Treated  through  a  process  cost  function,  to  be  included  as  a  constraint  in 
model  development. 

Error  Definitions/Taxonomy:  No  explicit  error  theory.  Errors  can  be  defined  by  model  developers  for  use  in 
individual  models. 

Risk  Computation/Mitigation  Methodology:  No  explicit  treatment  of  this  in  the  software. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Development  Steps  Supported: 

Function  analysis 
Function  allocation 
Task  design 

Interface  and  team  development 
Performance/workload/training  estimation 

How  Tool  Maps  System  Requirements  to  HSI  Requirements:  very  difficult  to  accomplish  this  mapping 
with  the  ACT-R  system.  The  high  degree  of  system  granularity  and  its  emphasis  on  purely  theoretical 
cognitive-perceptual  processes  prevents  easy  translation  to  HSI  or  system  requirements. 

Other  Evaluation  Criteria 

Learning  curve:  Steep 


Acquisition  Directorate 

Research  &  Development  Center 


B-6 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


Reliability,  Validity,  and  Platform  Requirements: 

ACT-R  the  theory  is  embodied  in  ACT-R  the  software,  as  a  set  of  functions  and  algorithms  implemented 
in  Common  Lisp.  Since  the  ACT-R  implementation  lives  in  Lisp,  the  aspiring  cognitive  modeler  must  also 
have  access  to  some  Lisp  environment,  or  use  the  standalone  version  of  the  ACT-R  Environment. 

Available  on  Windows,  Mac  (PowerPC),  and  Unix 

Analysis  Utilities  and  Interface  Support:  Support  available  via  email  and  ACT-R  user  group.  No  formal 
support. 

Availability,  Cost,  and  Contact  Information: 
http://act-r.psy.cmu.edu/actr6/ 

http://en.wikipedia.org/wiki/ACT  (cognitive  model  1 

B.3  Critical  Decision  Method  (CDM) 

Description 

Used  to  capture  critical  decisions  needed  in  managing  complex  work  domains,  the  environments  in  which 
decisions  are  made,  decision  dynamics  and  triggers,  common  errors,  tools  used  to  assist  in  decision-making 
(if  any),  and  the  outcomes  of  the  decisions. 

Tool/Method  Content 

Theoretical  Assumptions: 

•  Decisions  are  recognition-primed,  that  is,  they  rely  on  complex  recognition  of  conditions  and 
constraints  occurring  in  real  time.  They  are  not  analytical  and  deliberate,  as  assumed  by  many 
classical  models  of  decision-making. 

•  The  ability  to  make  rapid,  recognition-based,  effective  decisions  is  a  function  of  expertise  and, 
therefore,  arises  after  training  and  exposure  to  a  range  of  operational  situations. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

•  Some  implications  for  manpower  can  be  derived  indirectly  from  use  of  CDM  methods. 

Questions  Addressed: 

•  Decisions  required  for  successful  performance/effectiveness 

•  Some  notion  of  decision  criticality 

•  Decision  triggers  (often  expressed  as  perceptual  patterns) 

•  Common  errors  for  the  types  of  decisions  being  made 

•  Common  decision  strategies 

Content  Modeled: 

•  Information,  common  errors  and  environmental  factors  triggering  the  decisions. 
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Model  Granularity: 

•  Descriptive  only.  Provides  a  high-level  description  of  the  decision  “space”  and  factors  influencing 
decision  effectiveness. 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

•  Output  usually  rendered  as  a  spreadsheet  with  decisions  and  supporting/descriptive/forensic 
information.  Another  form  of  output  consists  of  a  detection-interpretation-control-feedback  diagram 
that  describes  decision  processes  tracked  through  time -based  or  sequence-based  stages.  An  example 
of  such  a  diagram  is  shown  below  (Figure  B-l). 

•  Most  terminology  taken  from  recognition-primed  decision  theory. 

•  Standard  elements  include  decisions,  triggers,  common  errors,  information  required  for  decision, 
perceptual  patterns  used  in  decision-making,  strategies. 
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Figure  B- 1 .  Critical  decision  method  diagram. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

•  No  explicit  discussion  of  using  critical  decision  theory  for  systems  engineering  design.  This  tool  most 
often  used  for  training  applications. 

•  Potential  utility  as  an  analytical  tool  contributing  to  requirements  definition  and  analysis. 
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Computer  Language  and  Interface  Support: 

•  None 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

•  No  discussion  of  risk  in  the  literature. 

Error  Definitions/Taxonomy: 

•  No  standard  or  well-defined  error  taxonomy.  Errors  tend  to  be  defined  uniquely  to  the  specific 
application  under  consideration. 

Risk  Computation/Mitigation  Methodology: 

•  None 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

•  Training  requirements 

•  Systems  engineering  requirements 

Development  Steps  Supported: 

•  Concept  definition 

•  Requirements  analysis 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

•  Required  and  critical  decisions  can  be  captured  in  system  requirements  and  mapped  to  HSI  domains, 
primarily  to  HFE. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

•  N/A 

Analysis  Utilities  and  Interface  Support: 

•  None 

Availability,  Cost,  and  Contact  Information: 

•  Freely  available 

•  Klein,  G.A.,  Calderwood,  R.  &  MacGregor,  D.  (1989).  Critical  decision  method  for  eliciting 
knowledge.  IEEE  Transactions  on  Systems,  Man  and  Cybernetics,  19(3). 
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B.4  Cognitive  Function  Analysis  (CFA) 

Description 

An  integrated  approach  to  human-centered  system  design.  Cognitive  functions  are  defined  as  a  mapping 
from  a  task  to  an  activity.  Tasks  are  things  that  a  system  user  is  required  to  do.  Activities  are  the  actions 
workers  perform  to  complete  a  task.  The  phases  of  this  approach  include  design  of  a  set  of  primitive 
cognitive  functions  through  the  use  of  participatory  design  and  domain  analysis,  definition  of  evaluation 
criteria  to  guide  the  distribution  of  cognitive  functions  among  agents,  and  incremental  design  and 
assessment  of  cognitive  functions  by  designers,  users,  and  usability  specialists  that  are  used  to  build  active 
design  documents.  Active  design  documents  (ADD)  provide  interactive  and  dynamic  explanations  about 
the  way  the  system  should  be  or  actually  is  used,  as  well  as  a  trace  of  the  design  rationale  as  a  function  of 
usability  criteria.  ADD  include  interaction  descriptions,  interface  objects,  and  contextual  links. 

Tool/Method  Content 

Theoretical  Assumptions: 

(1)  Notes  a  shift  from  energy-intensive  to  information- intensive  interaction. 

(2)  Defines  the  AUTO  pyramid:  Artifact,  User,  Task,  Organization.  The  axes  connecting  these  four 
anchors  indicate  the  information,  challenges  and  design  commitments  that  one  must  address  in  user- 
centered  design. 

(3)  Distinction  between  artifact-based  cognitive  function  transfer  and  task-based  cognitive  function 
transfer.  The  former  refers  to  automation  that  enhances  direct  manipulation.  The  latter  refers  to 
automation  enhancing  task  delegation  to  a  software  agent. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

Questions  Addressed: 

CFA  is  based  on  the  cognitive  systems  engineering  philosophy,  accordingly,  the  questions  addressed 
primarily  are  focused  on  task  requirements  in  context,  the  role  of  the  environment  in  shaping  task 
requirements  and  strategies,  constraints  and  adaptive  responding. 

Content  Modeled: 

Artifact,  user,  task,  organization,  procedure  training,  social  issues,  role  and  job  analysis,  task  and  user 
activities,  information  requirements,  technological  and  human  operator  limitations,  user  and  artificial 
cognition. 

Model  Granularity: 

Low.  Primarily  a  qualitative,  “paper  and  pencil”  method.  No  known  computer-based  development 
environment  that  would  force  careful,  consistent  definition  of  concepts  or  relationships.  At  best,  the 
method  seems  to  be  a  specification  only. 
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Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Accomplished  primarily  through  ADD.  ADD  are  defined  as  having  three  components: 

•  Interaction  descriptions:  Symbolically  conveys  ideas  and  information,  e.g.,  descriptions  of 
procedures.  The  interaction  descriptions  define  the  task  space. 

•  Interface  objects:  Constituents  of  interaction  descriptions,  these  contain  the  “emotive  aspects”  of 
the  design.  They  will  include  mockups  of  the  interface  being  designed.  Interface  objects  define 
the  activity  space. 

•  Contextual  links:  These  are  defined  as  the  “connective  tissue”  between  interaction  descriptions  and 
interface  objects.  Contextual  links  define  the  cognitive  function  space. 

used  to  Integrate  with  SE  and  Other  Environments: 

CFA  is  an  instance  of  participatory  design. 

No  explicit  way  to  convert  CFA  results  into  system  requirements. 

ADD  “are  shareable  prototypes  of  the  real  artifacts  being  designed  that  can  be  used  by  real  users  to 
assess  their  usability.”  This  statement  places  the  ADD  approach  somewhat  outside  the  traditional 
systems  engineering  requirements  development  process.  Integration  with  system  requirements 
would  be  an  indirect  effect  of  ADD  development. 

Their  use  of  ADDs  exists  more  at  the  specification  level  than  at  the  requirements  development 
level.  This  carries  the  danger  that  the  ADDs  “jump  over”  the  system  requirements  in  their  quest 
for  specifications,  thereby  creating  specs  that  can  be  disconnected  or  contradictory  to  the  system 
requirements. 

Computer  Language  and  Interface  Support: 

No  specific  computer  or  interface  support  for  CFA  beyond  the  specifications  and  some  structured  paper 
and  pencil  artifacts. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

•  No  explicit  treatment  of  risk. 

•  This  method  contains  the  basis  for  some  risk  analysis  associated  with  the  specifications  contained 
in  the  ADD.  However,  the  literature  contains  no  suggestion  that  this  area  has  been  developed. 

Error  Definitions/Taxonomy: 

No  defined  error  taxonomy. 

Risk  Computation/Mitigation  Methodology: 

•  No  explicit  treatment  of  risk. 

•  This  method  contains  the  basis  for  some  risk  analysis  associated  with  the  specifications  contained 
in  the  ADD.  However,  the  literature  contains  no  suggestion  that  this  area  has  been  developed. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Traceability  possible  only  with  an  explicit  linking  of  CFA  method  to  requirements  analysis.  This  is 
possible  in  principle,  however,  no  links  have  been  defined  in  the  literature. 


Methods 
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Development  Steps  Supported: 

•  Concept  definition 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Interface  and  team  development 

•  Training  development 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

No  explicit  method  for  accomplishing  this  mapping. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

•  Paper  and  pencil  method  only. 

•  No  known  reliability  or  validity  studies. 

•  No  specific  platform  requirements. 

Analysis  Utilities  and  Interface  Support: 

N/A 

Availability,  Cost,  and  Contact  Information: 

•  Freely  available 

•  www.eurisco.org 

References: 

Boy,  G.  A.  Cognitive  Function  Analysis.,  Stamford,  CT:  Ablex  Publishing  Corporation,  1998. 

Boy,  G.  A.  Active  Design  Documents  as  Software  Agents  that  Mediate  Participatory  Design  and 
Traceability.  In  Chipman,  Shalin  &  Schraagen,  Eds.  Cognitive  Task  Analysis.  New  Jersey:  Lawrence 
Erlbaum,  2000. 

B.5  Concept  Mapping 

Description 

Concepts  and  labeled  links  between  concepts  are  formed  into  networks  describing  work,  organizations, 
information  flows,  relationships  and  so  on.  Because  the  concept  mapping  is  atheoretical  and  general  in  its 
taxonomic  structure  it  can  be  used  to  describe  almost  any  design  problem.  Typically,  an  ontology  or 
taxonomy  is  developed  for  the  particular  application  being  addressed.  For  example,  typical  taxonomic 
categories  for  human  interaction  with  complex  systems  might  include  tasks,  activities,  organizations,  roles, 
products,  artifacts,  tools,  interfaces,  cognitive  work  elements,  information,  data,  relationships  and  so  on. 
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Tool/Method  Content 

Theoretical  Assumptions: 

This  method  is  essentially  atheoretical.  Its  only  assumption  is  that  the  content  of  analysis  will  consist  of 
concepts,  represented  as  boxes,  and  relations,  represented  as  links  between  concepts.  Links  can  be 
unidirectional  or  bidirectional.  Any  theory  is  added  by  the  analyst  in  the  form  of  ontological  definition. 

HSI  Domains  Addressed: 

Can  potentially  address  any  of  the  HSI  domains  due  to  the  flexibility  of  the  representational  system.  One 
must  simply  define  a  taxonomic  system  for  the  domain  being  analyzed. 

Questions  Addressed: 

The  open-ended  nature  of  concept  mapping  allows  analysts  and  designers  to  address  most  any  question 
needed,  assuming  they  can  define  an  analysis  system  that  can  represented  by  a  basic  “boxes  and  arrows” 
system. 

Content  Modeled: 

Any  content  needed. 

Model  Granularity: 

Can  be  at  whatever  level  the  designer  needs.  Decomposition  can  be  created  for  as  many  levels  as  desired. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Basic  concepts  and  links  between  concepts.  Taxonomic  conventions  are  defined  by  the  analyst.  Links  to 
external  documents  can  be  added  to  concepts  or  relations  contained  in  the  concept  maps.  Figure  B-2 
shows  a  master  concept  map  supporting  the  design  of  a  visualization-based  decision  support  system  for 
Air  Force  operational  assessment.  Node  colors  correspond  to  categories  relevant  to  a  cognitive  system 
engineering  analysis  (operational  processes,  tasks  and  activities,  products  and  artifacts,  requirements  and 
contingencies;  systems,  tools  and  interfaces;  behavioral  and  decision  processes  and  supporting 
information  and  data  needs;  organizations,  teams,  individuals  and  roles;  cognitive  and  perceptual 
requirements,  capabilities  and  limitations).  Links  between  nodes  express  relationships.  Analysts  would 
use  this  master  concept  map  (as  a  work  ontology)  to  generate  individual  concept  maps  capturing  all 
information  important  to  the  design  and  development  of  the  system  in  question  for  the  work  to  be 
supported  in  the  domain  of  interest. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

No  integration  method  as  part  of  the  concept  mapping  methodology.  Integration  method  is  completely 
analyst-determined. 

Computer  Language  and  Interface  Support: 

Several  concept  mapping  environments  are  available.  For  example: 
http://www.inspiration.com 

http://www.semanticresearch.com/products/semantica-pro.php 

http://cmap.ihmc.us/ 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

None 

Error  Definitions/Taxonomy: 

None 

Risk  Computation/Mitigation  Methodology: 
None 
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Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Specific  output  categories  are  up  to  the  designer  and  analyst.  Concept  maps  can  contribute  to  the 
development  of  CONOPS  and  early  requirements  documents.  They  can  contribute  to  requirements 
analysis  and  development  if  a  method  for  translating  their  content  to  a  form  compatible  with  requirements 
is  available. 

Development  Steps  Supported: 

•  Concept  definition 

•  Requirements  analysis 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Requirements  review 

•  Training  development 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Nothing  inherent  in  these  tools  to  do  this.  Mapping  across  these  domains  must  be  defined  and 
implemented  by  the  analyst. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

•  N/A 

•  There  are  concept  mapping  tools  available  for  most  common  computing  platforms. 

Analysis  Utilities  and  Interface  Support: 

•  Varies  by  tool.  Freeware  tools  have  little  or  no  support  for  tasks  beyond  the  creation  of  concepts  and 
links. 

•  Commercial  tools  often  have  some  analysis  tools  included,  particularly  in  the  area  of  network 
analysis  and,  in  some  cases,  some  reasoning  capability  over  the  network. 

Availability,  Cost,  and  Contact  Information: 

Tools  range  from  freeware  to  commercial  tools.  The  best  freeware  tool  available  is  the  IHMC  Cmap  tools 
available  from  the  Institute  for  Human  and  Machine  Cognition  at  the  University  of  West  Florida. 
http://cmap.ihmc.us/ 

B.6  Goals,  Operators,  Methods,  and  Selection  rules  (GOMS) 

Description 

GOMS  is  an  approach  to  human  computer  interaction  (HCI)  observation.  Goals  are  what  the  user  intends  to 
accomplish.  Operators  are  actions  that  are  performed  to  get  to  the  goal.  Methods  are  sequences  of  operators 
that  accomplish  a  goal.  There  can  be  more  than  one  method  available  to  accomplish  a  single  goal,  if  this  is 
the  case  than  selection  rules  are  used  to  describe  when  a  user  would  select  a  certain  method  over  the  others. 
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Types  of  GOMS: 

•  Keystroke-Level  Model  (KLM)  described  by  Card,  Morgan,  Newell.  Contains  several  simplifying 
assumptions. 

o  Uses  pre-established  keystroke-level  primitive  operator  for  predictions, 
o  Specified  method  is  limited  to  being  in  sequence  form  and  containing  only  keystroke  level 
primitive  operators. 

•  GOMS  developed  by  Card,  Morgan,  Newell  (CMN-GOMS). 

o  Slightly  more  specified  than  general  GOMS 

o  Hierarchical  goal  structure  and  methods  in  program  form.  Represented  in  pseudo-code-like 
notation  that  can  include  sub-methods  and  conditionals, 
o  Each  method  consists  of  a  series  of  steps  executed  in  strictly  sequential  order, 
o  One-to-one  correlation  of  physical  operator  in  CMN-GOMS  with  the  K’s  (press  Key  or 
button)  and  P’s  (Pointing  with  a  mouse)  of  KLM 
o  Puts  mental  time  in  “verify”  operators  at  end  of  sub-procedures. 

•  Natural  GOMS  Language  (NGOMSL) 

o  Provides  well-defined,  structured  natural  language 
o  NGOMSL  models  are  in  program  form 

o  Uses  breadth- first  expansion  of  user’s  top-level  goals  into  methods 
o  Can  be  used  to  estimate  learning  time  and  execution  time 
o  Assumed  knowledge  of  execution  of  operators 
o  Has  limitations  —  assumes  linear  tasks 

•  Cognitive-Perceptual-Motor  (Alternatively,  Critical  Path  Method)  (CPM-GOMS) 

o  Based  directly  on  the  parallel  multi-processor  stage  model  of  human  information  processing 
(Model  Human  Processor) 

o  No  human  assumptions  that  operators  are  performed  serially  —  Perceptual,  cognitive,  and 
motor  operators  at  the  level  of  Model  Human  Processor,  processor  cycle  times  can  be 
performed  in  parallel  as  the  task  demands 

o  Begins  with  a  CMN-GOMS  model,  and  starts  out  serially  then  interleaved  to  take  advantage 
of  parallelism 

o  Is  overly  detailed  for  serial  tasks 
o  Assumes  that  the  user  is  experienced 

o  Requires  understanding  of  parallel  processing  and  information- flow  dependencies 

Tool/Method  Content 

Theoretical  Assumptions: 

Skilled  behavior  is  organized  as  a  set  of  productions.  All  behavior  is  goal-directed.  When  a  system  reaches 
an  impasse  in  its  problem-solving  behavior,  it  will  decompose  the  substance  of  the  impasse  into  smaller 
problem  definitions  and  attempt  to  solve  these  smaller  problems.  Solutions  will  be  integrated  until  the 
original  problem  has  been  solved. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 
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Questions  Addressed: 

Systems  can  be  compared  to  one  another  to  see  how  performance  changes  or  is  different.  It  can  provide 
specific  details  about  time  to  do  tasks. 

Content  Modeled: 

GOMS  breaks  down  users  interactions  with  a  system  into  their  most  primitive  actions.  These  actions  can 
be  physical,  cognitive,  or  perceptual.  Traditionally  these  actions  are  based  around  the  use  of  a  software 
interface. 

Model  Granularity: 

Breaks  tasks  down  into  their  smallest  possible  pieces.  Granularity  can  be  adjusted  to  capture  what  the 
evaluator  wants  to  examine. 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  is  expressed  in  terms  of  model  elements  (goals,  operators,  methods  and  selection  rules),  task 
specifications  and  performance  data  under  varying  task  conditions.  Taxonomic  conventions  are  those  of 
goal-directed  processing,  declarative  and  procedural  knowledge,  forms  of  learning  that  lead  to  creation  of 
production  rules,  and  model  assumptions  about  performance  dynamics  associated  with  fundamental 
perceptual  and  cognitive  processes.  Figure  B-3  shows  a  comparison  of  two  cursor-control  interfaces,  with 
associated  component  completion  times. 


Figure  B-3.  Use  of  GOMS  to  compare  two  interfaces. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

The  primary  method  of  integration  is  the  input  of  human  performance  findings,  for  the  system  or 
subsystem  under  analysis,  into  an  overall  systems  engineering  analysis  and  design  process. 


Computer  Language  and  Interface  Support: 

GOMS  is  primarily  a  theoretically  motivated  modeling  and  analysis  methodology,  with  no  particular 
supporting  computer  language.  Several  systems  based  on  the  GOMS  framework  have  been  developed. 
The  interface  support  associated  with  these  systems  varies. 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

No  explicit  assumptions  for  risk.  No  explicit  error  model.  These  areas  are  left  up  to  the  definitions 
provided  by  the  modeler  for  each  specific  application. 

Error  Definitions/Taxonomy: 

None.  See  comment  above. 

Risk  Computation/Mitigation  Methodology: 

None.  See  comment  above. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Development  Steps  Supported: 

•  Task  design 

•  Interface  development 

•  Performance/workload/training  estimation 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

GOMS  is  designed  to  be  used  after  requirements  have  been  defined.  Therefore,  there  is  no  method 
resident  in  the  approach  itself  that  addresses  system  to  HSI  requirements  mapping. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

GOMS  is  not  a  software  application  in  of  itself,  but  some  groups  have  created  software  around  the  GOMS 
framework  and  concepts. 

Analysis  Utilities  and  Interface  Support: 

Most  software  resources  have  been  created  at  academic  institutions  and  are  not  warranted.  Interface 
support  is  minimal. 

Availability,  Cost,  and  Contact  Information: 

All  known  software  is  free  of  charge. 
http://www.cs.cmu.edu/~bei/cogtool/ 


Acquisition  Directorate 

Research  &  Development  Center 


B-18 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


B.7  iGEN 

Description 

iGEN  is  a  Cognitive  Agent  Software  Toolkit  that  allows  you  to  develop  your  own  applications  in-house. 

Use  iGEN  to  develop  software  solutions  that  emulate  human  decision-making  and  problem-solving  skills. 
iGEN  lets  you  build  intelligence  into  your  IT  infrastructure  and  process  automation,  whether  for  training, 
performance  support,  or  simulation.  By  emulating  human  decision-making  processes  and  problem-solving 
skills,  iGEN  helps  you  capture  knowledge  in  the  terms  your  people  understand  and  your  systems  use.  iGEN 
allows  you  to  put  that  knowledge  into  action,  improving  productivity  and  efficiency  through  better  support 
and  enhanced  training. 

iGEN  is  a  complete  Integrated  Development  Environment  (IDE)  for  building  embeddable  cognitive  agents. 

The  iGEN  cognitive  engine,  called  BATON,  is  an  implementation  of  a  broader  framework  for  modeling 
human  information  processing.  That  framework  is  described  in  the  research  literature  under  the  name 
COGNET. 

Tool/Method  Content 

Theoretical  Assumptions: 

iGEN  takes  the  idea  that  human  beings  are  the  best  example  of  intelligence  available  to  us,  and  applies  the 
cognitive  science  research  as  the  basis  for  tools  to  build  cognitive  agents.  Intelligent  human-like  software 
programs  that  think  like  humans  offer  many  potential  advantages  over  traditional  ‘dumb’  software  or 
conventional  AI: 

•  Their  actions  should  be  more  human-like  and  understandable  to  the  people  that  need  to  interact  with 
them; 

•  The  knowledge  the  agents  need  should  be  more  readily  obtainable  from  human  experts  in  the  same 
field  of  work;  and 

•  The  agent’s  internal  reasoning  and  thought  processes  should  be  easier  to  analyze  and  debug. 

Openness  and  extensibility:  The  iGEN  cognitive  engine  uses  an  architecture  based  on  cognitive  research, 
but  the  architecture  minimizes  the  number  of  ‘built-in’  psychological  theories.  As  a  result,  it  has  retained 
an  open  architecture,  which  allows  different  component-level  theories  (e.g.,  of  vision,  audition, 
grasp/reach,  memory  decay,  and  so  on)  to  be  built  and  inserted  into  specific  applications  as  needed  and 
desired  by  the  end-user. 

Scalability  and  compatibility  with  complex  expertise:  Unlike  the  highly  constrained  settings  in  which 
much  basic  cognitive  research  has  been  carried  out,  real-world  cognitive  agents  must  operate  in  large 
complex  problem  environments.  They  have  to  be  able  to  incorporate  the  sophisticated  strategies  used  by 
true  human  experts,  such  as  those  identified  by  research  on  naturalistic  decision  making.  iGEN  was 
deliberately  designed  to  be  compatible  with  these  expert  strategies,  through  its  use  of  pattern-directed 
attention  and  highly  chunked  goal  structures.  This,  in  turn,  has  given  iGEN  cognitive  agents  an  ability  to 
scale  up  to  very  complex  and  dynamic  applications  that  would  be  unapproachable  by  other  methods. 
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Separation  of  competence  and  performance:  There  are  many  limitations  of  human  cognition  that  an  agent- 
builder  won’t  want  to  replicate  in  a  cognitive  agent,  such  as  limited  processing  speed,  a  propensity  for 
errors,  a  memory  which  forgets  information,  and  so  on.  iGEN  differentiates  between  these  limiting  factors 
which  model  human  performance  and  the  overall  unconstrained  abilities  which  give  rise  to  human 
competence.  In  applications  such  as  decision  support  or  intelligent  tutoring,  the  cognitive  agent  developer 
typically  wants  the  advisory  or  tutorial  reasoning  within  the  agent  to  be  as  competent  as  possible.  On  the 
other  hand,  when  the  cognitive  agent  is  a  simulation  of  a  human  (such  as  an  equipment  operator),  then 
producing  realistic  human  performance  is  paramount.  The  iGEN  cognitive  engine  captures  expertise  in  a 
way  that  allows  unconstrained  execution  to  model  human  competence,  but  also  allows  specific 
performance-constraining  factors  to  be  incorporated  to  create  execution  as  a  human  performance  model. 

Support  for  multi-tasking  and  time-critical  applications:  Cognitive  agent  applications  typically  need  to 
help  (or  simulate)  people  who  need  to  make  time-critical  decisions  and  who  are  working  on  many  tasks 
simultaneously  (i.e.,  are  multi-tasking).  iGEN  was  designed  to  deal  with  competing  demands  for  attention 
and  a  constantly-changing  set  of  circumstances.  The  iGEN  cognitive  engine  uses  a  memory-centric, 
situation-based  attention  process  that  allows  the  cognitive  agent  to  be  highly  responsive  to  changing 
situations,  to  be  able  to  interrupt  itself  when  necessary,  and  to  prioritize  among  many  competing  demands 
on  the  basis  of  the  current  context. 


HSI  Domains  Addressed: 

•  HFE 

•  Manpower 

•  Personnel 

•  Training 

Questions  Addressed: 

A  significant  problem  for  training  individuals  and  teams  is  the  expense  associated  with  the  time  and 
resources  required  (e.g.,  support  personnel,  team  members,  instructors).  iGEN(R)  cognitive  agents  can  be 
used  in  different  ways  to  provide  a  better  training  environment. 

Synthetic  Teammates:  iGEN(R)  can  be  used  to  build  agents  that  synthetically  represent  team  members 
during  team  training.  These  synthetic  teammates  are  capable  of  standing  in  for  their  human  counterparts  to 
support  individual  or  partial-team  training  on  teamwork  skills. 

Synthetic  Instructors:  Synthetic  instructors  use  domain  and  task  knowledge  to  provide  context-based 
observations  and  assessments  of  trainee  behavior.  They  can  be  designed  to  provide  assessments  that  may 
be  diagnostic  at  either  the  behavioral  level  (e.g.,  trainee  did  not  perform  a  required  procedure)  or  at  the 
cognitive  level  (e.g.,  trainee  did  not  understand  the  required  procedure).  These  assessments  may  be 
presented  to  the  trainee  during  the  training  exercise  and-or  after  the  exercise  (i.e.,  after  action  review). 


Simulating  complex  systems  before  they  are  built  allows  detailed  non-destructive  and  low-cost  analysis 
and  testing.  Often,  though,  the  critical  component  of  simulation  is  the  human  element.  The  most  important 
questions  can’t  be  answered  through  simulation  without  a  reasonable  model  of  how  the  human  operator, 
human  team  or  human  organization  will  behave.  iGEN(R)  human  performance  models  (HPM)  provide  the 
modeling  and  simulation  community  with  a  powerful  new  tool  to  represent  the  human  element  in 
advanced  simulation. 
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Content  Modeled: 

iGEN(R)  HPM  agents  can  be  built  to  represent  the  knowledge-based  work  and  decision  processes  of 
individuals  (such  as  workstation  operators),  teams  (such  as  command  teams  or  control  room  teams),  or 
even  whole  organizations  (such  as  command  and  control  nodes,  enemy  or  competition  organizations). 
iGEN(R)  HPM  agents  incorporate  the  knowledge  and  work  processes  of  the  human  element  being 
simulated.  The  powerful  iGEN(R)  cognitive  agent  architecture  applies  these  to  simulate  the  desired  range 
of  behavior,  including  individual  variability,  errors,  population  variability,  and  some  forms  of  learning. 
The  result  is  dynamic  and  adaptive  behavioral  simulations  that  are  robust  and  scenario-independent. 

Model  Granularity: 

Because  iGEN  was  designed  to  support  as  large  a  range  of  applications  as  possible,  it  does  not  have  any 
‘least  common  denominator’  of  granularity  for  representing  knowledge  and-or  cognitive  processes.  Unlike 
other  architectures,  which  are  tied  to  fixed  cognitive  cycle  time  (typically  a  small  fraction  of  a  second),  the 
iGEN  cognitive  engine  allows  the  cognitive  agent  builder  to  select  a  level  of  detail  that  is  appropriate  for 
the  application  at  hand.  This  allows  knowledge  about  the  application  domain  to  be  programmed  at  the 
level  most  appropriate  for  the  needs  of  the  specific  application. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  includes  task  specifications,  performance  data  and  results,  and  information  about  errors  during 
operations.  Terminology  includes  standard  discrete-event  and  task  network  terms,  human  information 
processing  (HIP)  terms,  performance  parameters  associated  with  HIP  dynamics. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 
iGEN  has  convenient  implementation  features  that  make  it  attractive  for  incorporation  into  your 
development  environment: 

•  Workbench-based  development  approach — a  collection  of  high  level  agent-building  tools  that 
facilitate  engineering  of  intelligent  applications 

Computer  Language  and  Interface  Support: 

•  Visual  programming  interface — a  graphical  way  of  defining  program  logic  and  knowledge,  for 
easier  use  by  programmers  and  non-programmers,  alike 

•  Well-structured  Application  Program  Interface — permitting  integration  of  iGEN  cognitive  agents 
with  and  within  existing  applications  using  standard  languages  (e.g.,  C/C++,  Java)  and  protocols 
(COM,  CORBA,  HLA) 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risk  can  be  modeled  in  iGEN  by  defining  relationships  between  errors  committed  by  operators  and 
resulting  consequences  on  system  performance/effectiveness.  Modelers  define  these  relationships  for  the 
systems  and  operational  contexts  under  consideration. 

Error  Definitions/Taxonomy: 

iGEN  does  not  contain  error  categories  as  part  of  the  software  package.  HS  analysts  can  define  errors  as 
needed  for  particular  modeling  applications. 
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Risk  Computation/Mitigation  Methodology: 

No  risk  computation  process  included  with  the  software.  The  system  can  be  used  to  produce  models  that 
allow  analyses  of  performance  under  various  operating  conditions,  thereby  producing  data  that  inform 
separate  risk  analyses. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Model  output  includes  operator  performance,  efficiency  estimates,  error  characteristics  (if  defined),  and 
temporal  dynamics  (e.g.,  time  to  proficiency)  that  can  be  integrated  with  other  acquisition  and  system 
requirements. 

Development  Steps  Supported: 

•  Requirements  analysis 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Interface  and  team  development 

•  Performance/workload/training  estimation 

•  Requirements  review 

•  Training  development 

•  Performance  assurance 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

No  facilities  within  the  tool  to  do  this.  However,  because  iGEN  supports  rigorous  simulation  of  human 
performance  levels  and  dynamics,  HSI  requirements  can  be  derived  from  simulations.  These  then  can  be 
integrated  with,  or  compared  to,  system  requirements. 

Other  Evaluation  Criteria 

Learning  curve:  Steep 

Reliability,  Validity,  and  Platform  Requirements: 
iGEN  IDE  System  Requirements  (Version  2.0): 

•  Windows  95,  98,  2000,  NT,  XP 

•  We  recommend  at  least  64  Mb  RAM,  recommend  over  256  MB 

•  Build  the  required  communication  shell  (API),  using  Microsoft  Visual  C++  6.0,  service  pack  3  or 
greater  for  Windows  applications,  or  gcc  3.2  or  greater  for  Linux  and  other  applications,  including 
embedded  systems. 

Standalone  Agent  Requirements: 

•  Windows  (95,  98,  2000,  NT,  XP),  Unix  (Linux  (RedHat  6.1+),  IRIX  (6.5),  Solaris  (2.6)) 

•  Recommend  >256  MB  of  RAM 

Analysis  Utilities  and  Interface  Support: 

Delivered  as  part  of  the  standard  software  package  and  user  support  license. 
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Availability,  Cost,  and  Contact  Information: 
http://www.chisystems.com 

Jim  Hicinbothom,  iGEN(R)  Product  Manager  /  858-618-1060 
Email:  jhicinbothom@chisvstems.com  or  iGEN@chisystems.com 


B.8  Information  and  Functional  Flow  Analysis 

Description 

In  information  flow  analysis,  a  flowchart  of  the  information  and  decisions  required  to  satisfy  the  functions 
of  a  complex  system  is  constructed.  These  charts  describe  information  flow  and  highlight  critical  paths  and 
bottlenecks  as  information  flows  among  nodes  in  the  system. 

In  functional  flow  analysis  the  system  is  decomposed  into  the  functions  it  must  support.  Function-flow 
diagrams  are  constructed  to  show  the  sequential  or  information-flow  relationships  between  system 
functions.  Petri  nets  may  be  used  as  a  modeling  formalism  to  implement  function-flow  diagrams. 


Tool/Method  Content 

Theoretical  Assumptions: 

In  system  dynamics,  flows  represent  information,  products  and  processes  of  interest.  They  alter  the  state 
of  a  system  by  altering  component  states  of  system  constituents.  Since  flows  are  the  rates  at  which  system 
states  change,  they  can  be  used  to  assess  the  performance  levels  of  a  system  of  interest  by  deriving 
measures  such  as  capacity,  flow  amounts,  flow  rates  and  other  properties. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

•  Manpower 

Normally,  flow  analysis  is  used  in  analysis  and  evaluation  of  issues  involving  HFE  and  training  concerns 
in  system  development.  However,  because  the  theory  underlying  flow  analysis  is  so  flexible,  the  method 
can  be  used  in  any  domain  in  which  system  states  and  flow  dynamics  can  be  consistently  defined. 

Questions  Addressed: 

Fundamentally,  the  questions  addressed  by  all  types  of  flow  analysis  involve  the  computation  of  system 
state  as  a  function  of  the  flow  dynamics  that  apply  to  the  system.  Some  examples  of  specific  questions 
that  can  be  addressed  include: 

•  Performance  levels  as  a  function  of  information  arrival  rates, 

•  Performance  as  a  function  of  information  carrying  or  processing  capacity, 

•  Performance  as  a  function  of  average  throughput  in  a  system, 

•  Performance  as  a  function  of  disequilibria  involving  information  flow  through  a  system. 


Content  Modeled: 

Flow  analyses  model  functions  (or  processes)  and  information  in  a  system,  along  with  the  dynamics  of 
these  concepts.  As  stated  above,  dynamics  can  include  capacities,  rates,  accumulations,  delays, 
disequilibria,  and  other  dynamics  related  to  system  performance  and-or  effectiveness.  The  specific 
content  of  a  model  or  analysis  will  depend  on  the  system  under  study  and  the  questions  being  asked  by  the 
development  team. 
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Model  Granularity: 

The  models  typically  developed  under  flow  analyses  will  exist  at  the  system  level.  Because  these  model 
are  functional  in  nature,  we  consider  this  to  represent  a  mid-level  of  granularity.  Detailed  processes,  such 
as  those  articulated  by  analysis  systems  like  ACT-R,  are  not  addressed  with  flow  analysis. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Flow  analysis  uses  the  terminology  of  system  dynamics:  processes  and  flows  (or  some  equivalent 
nomenclature),  system  state,  transfer  functions,  accumulations  (or  integral  notation),  rates  (or  some  other 
derivative  notation),  delays,  and  disequilibrium  dynamics.  The  forms  of  output  can  be  either  descriptive 
or  formal.  For  purposes  of  system  acquisition  and  development  the  form  of  output  can  be  stated  in  terms 
of  requirements  and  system  performance  constraints. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

The  system  orientation  of  flow  analysis  can  be  used  to  integrate  with  other  systems  engineering  activities 
during  the  early  stages  of  system  design  and  development.  In  general,  flow  analysis  can  be  integrated 
with  any  other  development  activity  which  relies  on  the  language  of  system  dynamics  in  solving  design 
and  development  problems. 

Computer  Language  and  Interface  Support: 

There  seem  to  be  very  few  tools  available  that  are  specifically  devoted  to  flow  analysis  applied  to  HSI 
domain  issues.  One  tool  formerly  available  was  the  Information  Flow  Analysis  Software  Tool  (IF AST), 
in  development  for  a  time  by  Aptima  and  Micro- Analysis  and  Design  (now  a  division  of  Alion  Science 
and  Technology).  This  was  to  be  a  model-driven  software  tool  to  represent  and  analyze  information  flow 
in  complex  systems.  IF  AST  was  a  simulation-based  software  tool  to  analyze  information  flow,  highlight 
critical  paths  and  bottlenecks,  and  answer  “what-if  ’  questions  regarding  complex  human-machine 
systems.  It  included  the  capability  to  represent  the  human  decision  maker,  resulting  in  more  representative 
information  flow  metrics  for  analyses  relevant  to  HSI  concerns.  IFAST  produced  metrics  of  information 
flow  by  providing  users  with  a  simulation  capability  tailored  towards  the  flow  of  messages.  This  tool 
apparently  is  no  longer  available. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

All  risk  assumptions  in  flow  analysis  are  developed  by  the  analyst,  tailored  to  the  needs  of  the  specific 
system  under  development. 

Error  Definitions/Taxonomy: 

No  explicit  error  definition  taxonomy.  Derivative  taxonomies  can  be  developed  based  on  the  system 
dynamics  concepts  of  the  flow  analysis  methodology. 

Risk  Computation/Mitigation  Methodology: 

No  explicit  risk  computation  method.  Derivative  methods  can  be  developed  based  on  the  system 
dynamics  concepts  of  the  flow  analysis  methodology. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 
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Outputs  consist  of  system  states  and  information  flow  dynamics.  This  information  is  useful  in 
requirements  and  function  analysis.  It  also  informs  functional  allocation  decisions,  task  design  and 
performance  estimation. 

Development  Steps  Supported: 

•  Requirements  analysis 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Performance/workload/training  estimation 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

The  system  representations  provided  by  the  flow  analysis  methodology  of  these  tools  provides  a  way  to 
represent  needed  system  dynamics,  as  articulated  in  the  system  requirements,  against  the  capabilities  and 
limitations  of  human  components  of  the  system.  This  allows  analysts  to  model  the  system  specified  by 
these  requirements  and  evaluate  its  effectiveness,  attributing  shortfalls  to  human  and  non-human  system 
components.  In  cases  in  which  shortfalls  can  be  attributed  to  humans,  appropriate  HSI  requirements  can 
be  defined  or  modified. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

The  flow  analysis  methodology  has  been  shown  to  be  both  valid  and  reliable  across  a  wide  range  of 
applications.  There  are  no  particular  platform  requirements  for  the  use  of  this  methodology. 

Analysis  Utilities  and  Interface  Support: 

No  analysis  utilities  other  than  those  associated  with  general  system  theory  and  analysis.  There  are  no 
software  tools  for  flow  analysis  known  at  this  time,  therefore,  no  interface  support. 

Availability,  Cost,  and  Contact  Information: 

The  flow  analysis  methodology  is  widely  available  from  many  sources 

Davis,  J.  G.,  Subrahmanian,  E.,  Konda,  S.,  Granger,  H.,  Collins,  M.,  &  Westerberg,  A.  W.  Creating 
Shared  Information  Spaces  to  Support  Collaborative  Design  Work. 

Meister,  D.  (1989).  Conceptual  Aspects  of  Human  Factors.  Baltimore,  MD:  The  Johns  Hopkins  University 
Press. 

Randel,  J.  M.,  Pugh,  L.  H.,  &  Wyman,  B.  G.  Methods  for  Conducting  Cognitive  Task  Analysis  for  a 
Decision  Making  Task.  Navy  Personnel  Research  and  Development  Center  (1996).  The  Critical  Incident 
Method,  Interviewing,  and  Information  Flow  Modeling  are  used  to  study  a  decision  making  task. 

Kirwan,  B.  &  Ainsworth,  L.  K.  A  Guide  to  Task  Analysis.  London:  Taylor  and  Francis,  1992.  A  text  book 
of  various  task  analysis  techniques  and  their  use  in  the  systems  engineering  process. 
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B.9  Applied  Cognitive  Task  Analysis  (ACTA) 

Description 

A  method  for  performing  Cognitive  Task  Analysis.  The  method  consists  of  three  structured  interviews.  The 
first  interview  generates  a  Task  Diagram,  which  provides  a  broad  overview  of  the  task  and  highlights 
difficult  cognitive  portions  of  the  task  that  should  be  probed  further.  This  is  followed  by  a  Knowledge 
Audit,  which  surveys  the  aspects  of  expertise  required  for  a  specific  task  or  subtask.  Finally,  in  the 
Simulation  Interview  step,  the  cognitive  processes  of  experts  are  probed  within  the  context  of  a  specific 
scenario.  The  output  of  the  process  is  a  Cognitive  Demands  Table,  which  presents  the  results  so  they  can  be 
applied  to  a  specific  project. 

ACTA  is  an  instructional  software  tool  that  is  designed  to  assist  practitioners  in  identifying  cognitive  skills, 
or  mental  demands,  that  are  needed  to  perform  a  task.  These  skills/demands  include:  critical  cues  and 
patterns  of  cues;  assessment,  problem  solving,  and  decision-making  strategies;  why  these  are  difficult  for 
novices;  and  common  novice  errors.  ACTA  provides  a  means  for  practitioners  to  elicit  this  kind  of 
information  and  incorporate  it  into  instructional  design  interventions. 

Tool/Method  Content 

Theoretical  Assumptions: 

(1)  Experts  are  able  to  articulate  task  structure,  cognitive  requirements  and  critical  decisions  in  ways  that 
allow  designers  to  collect  the  information  needed  to  inform  system  design. 

(2)  Expertise  is  expressed  as  a  focused  recognition  of  critical  information  and  patterns  accompanied  by  an 
application  of  strategic  behaviors  in  the  service  of  goal  attainment. 

(3)  Knowledge  categories  critical  to  the  definition  and  development  of  systems  include  diagnosing  and 
predicting,  situation  awareness,  perceptual  skills,  developing  and  knowing  when  to  apply  specific 
strategies,  improvising,  recognizing  anomalies,  and  compensating  for  equipment  limitations.  Experts 
are  able  to  describe  their  task  requirements  in  terms  of  these  categories. 

(4)  Probes  can  be  developed  to  carry  out  audits  of  the  knowledge  needed  to  execute  critical  tasks  in  the 
design  problem  of  interest. 

HSI  Domains  Addressed: 

•  HFE 

•  Training. 

•  Personnel 

Some  of  the  information  acquired  by  this  method  can  be  used  in  safety  analyses,  although  this  use  would 
be  primarily  tangential. 

Content  Modeled: 

•  Operational  task  structure 

•  Strategic  performance  in  dynamic  task  circumstances 

•  Critical  decisions  and  the  task  contexts  in  which  these  arise 

•  Cognitive  content  in  task  performance 

•  Common  errors  committed  during  task  performance 
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Model  Granularity: 

To  the  extent  that  models  are  developed  based  on  the  information  gathered  with  ACTA,  granularity  is 
considered  to  be  low.  Most  of  the  techniques  used  under  the  ACTA  framework  are  paper  and  pencil,  and 
interview,  techniques.  The  procedures  used  here  are  intentionally  high-level  so  as  to  avoid  intractability 
and  a  reliance  on  “degenerating  into  a  search  for  everything  in  a  person’s  head.” 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  exists  primarily  in  the  form  of  tabular  and  survey  text  gathered  according  to  the  methodology 
shown  below. 


Figure  B-4.  Method  for  applied  cognitive  task  analysis. 

Some  Likert-type  data  can  be  collected,  if  desired,  but  these  data  will  be  subjective  in  nature.  Output 
terminology  follows  standard  cognitive  systems  engineering  conventions.  There  have  been  no  standard 
taxonomic  conventions  developed  for  this  tool.  Typical  output  is  shown  in  the  tables  below. 
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Table  B-l.  Example  simulation  interview  table.  (Source:  Militello  and  Hutton,  2000.) 


Events 

Actions 

Assessment 

Critical  Cues 

Potential  Errors 

On  scene 
arrival 

Account  for  people 
(names) 

Ask  neighbours 

Must  knock  on  or  knock 
down  doors  to  make 
sure  people  aren’t 
there 

It’s  a  cold  night; 
need  to  find  place 
for  people  who 
have  been 
evacuated 

Night  time 

Cold  >  15° 

Dead  space 

Add  on  floor 

Poor  materials,  metal  girders 

Common  attic  in  whole  building 

Not  keeping  track  of 
people  (could  be 
looking  for  people 
who  are  not  there) 

Initial 

attack 

Watch  for  signs  of 
building  collapse 

If  signs  of  building 
collapse,  evacuate 
and  throw  water  on  it 
from  outside 

Faulty 

construction; 
building  may 
collapse 

Signs  of  building  collapse  include: 

What  walls  are  doing:  cracking 

What  floors  are  doing:  groaning 

What  metal  girders  are  doing:  clicking, 
popping 

Cable  in  old  buildings  hold  walls  together 

Ventilating  the  attic; 
this  draws  the  fire 
up  and  spreads  it 
through  the  pipes 
and  electrical 
system 

Table  B-2.  Example  cognitive  demands  table.  (Source:  Militello  and  Hutton,  2000.) 


Difficult 

cognitive 

element 

Why  difficult? 

Common  errors 

Cues  and  strategies  used 

Knowing 
where  to 
search  after 
an  explosion 

Novices  may  not  be 
trained  in  dealing  with 
explosions.  Other 
training  suggests  you 
should  start  at  the  source 
and  work  outward. 

Novice  would  be  likely  to 
start  at  the  source  of  the 
explosion.  Starting  at  the 
source  is  a  rule  of  thumb 
for  most  other  kinds  of 
incidents. 

Start  where  you  are  most  likely  to  find  victims,  keeping  in 
mind  safety  considerations. 

Refer  to  material  data  sheets  to  determine  where 
dangerous  chemicals  are  likely  to  be. 

Consider  the  type  of  structure  and  where  victims  are 
likely  to  be. 

Consider  the  likelihood  of  further  explosions.  Keep  in 
mind  the  safety  of  your  crew. 

Finding 
victims  in  a 
burning 
building 

There  are  lots  of 
distracting  noises.  If  you 
are  nervous  or  tired,  your 
own  breathing  makes  it 
hard  to  hear  anything 
else. 

Novices  sometimes  don’t 
recognise  their  own 
breathing  sounds;  they 
mistakenly  think  they  hear 
a  victim  breathing. 

Both  you  and  your  partner  stop,  hold  your  breath  and 
listen. 

Listen  for  crying,  victims  talking  to  themselves,  victims 
knocking  things  over,  etc. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

There  are  no  specific  integration  methods  inherent  in  this  tool.  The  primary  method  of  integration  is  the 
presence  of  the  cognitive  systems  engineer  with  other  members  of  the  design  team.  ACTA  is  designed  to 
produce  information  that  can  be  incorporated  into  ongoing  system  requirements  development.  Successful 
incorporation  will  depend,  however,  on  satisfactory  translation  of  the  cognitive  task  analysis  information 
into  systems  engineering  and  requirements  language. 

Computer  Language  and  Interface  Support: 

Interface  support  will  consist  of  commonly  available  office  support  tools  and  databases. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

No  explicit  assumptions  regarding  risk. 
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Error  Definitions/Taxonomy: 

No  standard  error  taxonomy.  Error  definitions  are  left  up  to  the  individual  analyst. 

Risk  Computation/Mitigation  Methodology: 

No  standard  risk  computation  or  mitigation  methodology.  The  analyst  can  define  these  concepts  as  part  of 
the  ACTA  process  and  can  develop  risk  computations,  although  this  is  not  typically  done. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Output  categories  are  primarily  focused  on  cognitive  strategies,  decisions,  cognitive  errors.  This 
information  is  important  for  requirements  development  and  interface/interaction  specifications.  Much  of 
the  information  produced  by  this  tool  is  relevant  to  work  design,  error  identification  and  training.  Some 
information  might  be  related  to  safety  and  personnel  selection  considerations,  although  these  are  not 
explicitly  treated  by  the  tool  methodology. 

Development  Steps  Supported: 

•  Concept  definition 

•  Requirements  analysis 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Interface  and  team  development 

•  Training  development 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

No  explicit  mapping.  The  information  produced  by  the  tool  can  be  used  to  relate  system  requirements  to 
HFE  and  training  requirements.  The  method  of  doing  this  will  be  the  responsibility  of  the  individual 
analyst. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

The  ACTA  tool  was  validated  in  a  series  of  studies  carried  out  by  Militello,  et  al.  (1997).  The  reliability 
of  the  tool  has  not  been  established  across  multiple  domains  or  with  multiple  analysts.  There  are  no 
platform  requirements  associated  with  this  tool  other  than  those  needed  for  office  and  database  support. 

Analysis  Utilities  and  Interface  Support: 

Interface  support  will  consist  of  commonly  available  office  support  tools  and  databases. 

Availability,  Cost,  and  Contact  Information: 

Militello,  L.G.,  Hutton,  R.J.B.,  Pliske,  R.M.,  Knight,  B.J.,  &  Klein,  G.  (1997),  “Applied  Cognitive  Task 
Analysis  (ACTA)  Methodology,”  Fairborn,  OH:  Klein  Associates,  Inc.  Final  Technical  Report  prepared 
for  the  Navy  Personnel  Research  and  Development  Center  under  Contract  No.  N66001-94-C-7034. 
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B.10  Autonomous  Vehicle  Operator  Span  of  Control  Evaluation  Tool  (AVOSCET) 

Description 

AVOSCET  is  a  trade-off  analysis  tool  specifically  designed  to  help  analysts  determine  how  many 
autonomous  systems  an  operator  or  a  crew  can  control  under  a  variety  of  conditions. 

Tool/Method  Content 

Theoretical  Assumptions: 

Human  performance  can  be  described  as  a  set  of  networked  tasks,  modified  by  specifications  of 
capabilities  and  limitations  describing  operator  competencies.  When  situated  in  a  description  of  system 
dynamics,  operator  performance  can  be  used  to  moderate  overall  system  effectiveness. 

HSI  Domains  Addressed: 

•  HFE 

•  Safety 

•  Manpower 

Questions  Addressed: 

This  tool  is  focused  on  the  question  of  how  many  autonomous  vehicles  a  single  operator  can  control 
concurrently. 

Content  Modeled: 

Task  structure,  vehicle  dynamics,  environmental  conditions,  performance  moderators  and  stressors, 
temporal  requirements  and  characteristics. 

Model  Granularity: 

This  system  has  a  moderate  to  high  level  of  granularity.  Operators  can  be  modeled  in  detail  using  task 
performance  specifications  and  information  on  capabilities  and  limitations.  Environmental  and  task 
stressors  and  other  moderators  can  be  included  in  analyses.  It  is  not  possible  to  model  individual 
processes  in  this  system. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  system  presents  a  network  graph  of  critical  tasks,  as  shown  in  Figure  B-5.  Momentary  workload 
levels  as  a  function  of  many  inputs,  operator  performance  as  a  function  of  time,  stressors,  and 
environmental/vehicle  characteristics  are  then  computed  based  on  the  task  network  and  associated 
specifications,  as  shown  in  Figure  B-6. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Integration  is  accomplished  primarily  through  the  production  of  analysis  results  that  can  be  input  to  other 
systems  engineering  design  and  analysis  activities.  There  is  no  direct  means  of  integrating  into  other 
systems  engineering  tools. 
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Figure  B-5.  Part  of  an  AVOSCET  task  network, 
(from  www.maad.com) 
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■  Overall 
•  Visual 
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Figure  B-6.  AVOSCET  workload  analysis  display, 
(from  www.maad.com) 
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Computer  Language  and  Interface  Support: 

AVOSCET  is  based  on  the  MicroSAINT  discrete  event  simulation  package.  That  environment  is  based 
on  C#. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

No  explicit  risk  assumptions.  Any  risk  analyses  or  trade-offs  are  based  on  the  specifications  built  into  the 
simulation  by  the  analyst. 

Error  Definitions/Taxonomy: 

AVOSCET  contains  no  error  terminology. 

Risk  Computation/Mitigation  Methodology: 

The  system  contains  no  risk  computation  or  mitigation  support. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

AVOSCET  outputs  information  on  overall  operator  task  performance,  overall  effectiveness  within  the 
defined  operational  envelope,  and  workload  estimates. 

Development  Steps  Supported: 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Interface  and  team  development 

•  Performance/workload/training  estimation 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Analyses  can  be  used  to  constrain  system  requirements  in  the  areas  of  workload;  overall  task  performance; 
dynamic  task  performance  as  a  function  of  stressors,  moderators,  environmental  and  system 
characteristics;  and  manpower. 

Other  Evaluation  Criteria 

Learning  curve:  Steep 

Reliability,  Validity,  and  Platform  Requirements: 

No  data  have  been  found  regarding  reliability  and  validity.  Platform  requirements  for  this  tool  will  be 
based  on  those  for  the  MicroSAINT  simulation  environment,  i.e.,  Windows  platform. 

Analysis  Utilities  and  Interface  Support: 

The  tool  has  analysis  support  for  task  network  modeling  and  performance-based  analyses  of  human 
control  and  decision  making. 

Availability,  Cost,  and  Contact  Information: 

This  tool  is  commercially  available.  Cost  depends  on  the  number  of  licenses.  AVOSCET  was  developed 
by  Micro  Analysis  and  Design,  which  is  now  part  of  Alion  Science  and  Technology  Corp. 
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Alion  MA&D  Operation 
Corporate  Headquarters 
4949  Pearl  E.  Circle,  Suite  300 
Boulder,  CO  80301 
303-442-6947 
www.maad.com 

Stacey  Quesada,  squesada@alionscience.com 


B.ll  Man-Machine  Integrated  Design  and  Analysis  System  (MIDAS) 

Description 

The  Man-machine  Integration  Design  and  Analysis  System  (MIDAS)  is  a  three-dimensional  rapid 
prototyping  human  performance  modeling  and  simulation  environment  that  facilitates  the  design, 
visualization,  and  computational  evaluation  of  complex  man-machine  system  concepts  in  simulated 
operational  environments. 

MIDAS  combines  graphical  equipment  prototyping,  dynamic  simulation,  and  human  performance  modeling 
with  the  aim  to  reduce  design  cycle  time,  support  quantitative  predictions  of  human-system  effectiveness, 
and  improve  the  design  of  crew  stations  and  their  associated  operating  procedures. 

MIDAS  links  a  virtual  human,  comprised  of  a  physical  anthropometric  character,  to  a  computational 
cognitive  structure  that  represents  human  capabilities  and  limitations.  The  cognitive  component  is  made  up 
of  a  perceptual  mechanism  (visual  and  auditory),  memory,  a  decision  maker  and  a  response  selection 
architecture  (MicroSAINT  Sharp).  The  complex  interplay  among  bottom-up  and  top-down  processes 
enables  the  emergence  of  unforeseen,  and  non-programmed  behaviors. 

MIDAS  outputs  include  dynamic  visual  representations  of  the  simulation  environment,  timelines,  task  lists, 
cognitive  loads  along  six  resource  channels,  actual/perceived  situation  awareness,  and  human  error 
vulnerability  and  human  performance  quality. 


Tool/Method  Content 

Theoretical  Assumptions: 

MIDAS  assumes  that  the  human  operator  can  perform  multiple,  concurrent  tasks,  subject  to  available 
perceptual,  cognitive,  and  motor  resources.  MIDAS  is  an  integrative,  versatile  model  with  much  detail.  Its 
sub-models  are  often  based  on  current  psychological  can  psychomotor  theory  and  data.  Its  task  loading 
model  is  consistent  with  multiple  resource  theory.  MIDAS  explicitly  models  communication.  Much 
modeling  attention  has  been  given  to  situation  awareness  with  respect  to  the  updateable  world 
representation. 


HSI  Domains  Addressed: 

•  HFE 

•  Training 


Questions  Addressed: 

Over  its  history,  NASA’s  MIDAS  has  compared  human  performance  in  the  nuclear  power  plant, 
emergency  control  room  response  to  91 1  calls,  suit  design,  helicopter,  aviation,  and  space  domains. 
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Content  Modeled: 

The  overall  architecture  of  MIDAS  is  comprised  of  a  user  interface,  an  anthropometric  model  of  the 
human  operator,  symbolic  operator  models,  and  a  world  model.  MIDAS  is  an  object-oriented  system 
consisting  of  objects  (grouped  by  classes).  Objects  perform  processing  by  sending  messages  to  each  other. 

There  are  two  types  of  physical  component  agents  in  MIDAS:  equipment  agents  are  the  displays  and 
controls  with  which  the  human  operator  interacts;  physical  world  agents  include  terrain  and  aeronautical 
equipment  (such  as  helicopters).  Physical  component  agents  are  represented  as  finite-state  machines,  or 
they  can  be  time-script-driven  or  stimulus-response-driven.  Their  behaviors  are  represented  using  Lisp 
methods  and  associated  functions. 

The  human  operator  agents  are  the  human  performance  representations  in  MIDAS  —  cognitive, 
perceptual,  and  motor.  The  MIDAS  physical  agent  is  Jack™,  an  animated  mannequin.  MIDAS  uses  Jack 
to  address  workstation  geometry  issues,  such  as  the  placement  of  displays  and  controls.  Jack  models  the 
operator’s  hands,  eye,  and  feet,  though  in  the  MIDAS  version,  Jack  cannot  walk. 

The  visual  perception  agent  computes  eye  movements,  what  is  imaged  on  the  retina,  peripheral  and  foveal 
fields  of  view,  what  is  in  and  out  of  focus  relative  to  the  fixation  plane,  preattentional  phenomena  (such  a 
color  and  flashing),  detected  peripheral  stimuli  (such  as  color),  and  detailed  information  perception. 

Model  Granularity:  Unknown 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

In  a  MIDAS  simulation,  declarative  and  procedural  information  about  the  mission  and  equipment  is  held 
in  the  updatable  world  representation.  Information  from  the  external  world  is  filtered  by  perception,  and 
the  updatable  world  representation  is  updated.  Mission  goals  are  decomposed  into  lower-level  activities  or 
tasks,  and  these  activities  are  scheduled.  As  the  activities  are  performed,  information  is  passed  to  Jack, 
whose  actions  affect  cockpit  equipment.  The  external  world  is  updated,  and  the  process  continues. 

MIDAS  outputs  include  dynamic  visual  representations  of  the  simulation  environment,  timelines,  task 
lists,  cognitive  loads  along  6  resource  channels,  actual/perceived  situation  awareness,  and  human  error 
vulnerability  and  human  performance  quality. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Unknown 

Computer  Language  and  Interface  Support: 

MIDAS  is  written  in  the  Lisp  C,  C++  programming  languages  and  runs  on  Silicon  Graphics,  Inc., 
workstations.  It  consists  of  approximately  350,000  lines  of  code  and  requires  one  or  more  workstations  to 
run  on.  It  is  30  to  40  times  slower  than  real  time,  but  can  be  simplified  so  it  can  run  at  nearly  real  time. 

The  user  interface  consists  of  an  input  side  (an  interactive  Graphical  User  Interface  (GUI)),  a  cockpit 
design  editor,  an  equipment  editor,  a  vehicle  route  editor,  and  an  activity  editor.  On  the  output  side  there’s 
display  animation  software,  run-time  data  graphical  displays,  summary  data  graphical  displays,  and  3D 
graphical  displays. 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Many  MIDAS  behaviors,  such  as  operator  errors,  are  not  emergent  features  of  the  model,  but  must  be 
explicitly  programmed.  The  Z-Scheduler  makes  assumptions  that  are  somewhat  controversial.  The  scale- 
up  of  the  original  MIDAS  to  multi-operator  systems  would  appear  to  be  quite  difficult.  MIDAS  is  also  too 
big  and  too  slow  for  most  military  simulation  applications.  In  addition,  it  is  very  labor-intensive,  and  it 
contains  many  details  and  features  not  needed  in  military  simulations. 

Nevertheless,  MIDAS  has  a  great  deal  of  potential  for  use  in  military  simulations.  The  MIDAS 
architecture  would  provide  a  good  base  for  a  human  behavior  representation.  Components  of  MIDAS 
could  be  used  selectively  and  simplified  to  provide  the  level  of  detail  and  performance  required. 
Furthermore,  MIDAS  would  be  a  good  test-bed  for  behavioral  representation  research. 

Error  Definitions/Taxonomy: 

No  data. 

Risk  Computation/Mitigation  Methodology: 

No  data. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Primary  output  category  is  operator  performance.  MIDAS  supports  operator  requirements  analysis  and 
workload/performance/training  estimation. 

Development  Steps  Supported: 

•  Concept  definition 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Performance/workload/training  estimation 

•  Training  development 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Mapping  achieved  by  showing  relationships  between  operator  performance  and  system  effectiveness. 

Other  Evaluation  Criteria 

Learning  curve:  Steep 

Reliability,  Validity,  and  Platform  Requirements: 

MIDAS  is  written  in  the  Lisp,  C,  and  C++  programming  languages  and  runs  on  Silicon  Graphics,  Inc., 
workstations. 
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The  original  version  of  MIDAS  has  been  validated  in  at  least  one  experiment  involving  human  subjects. 
MIDAS  was  programmed  to  model  the  flight  crew  of  a  Boeing  757  aircraft  as  they  responded  to  descent 
clearances  from  air  traffic  control:  the  task  was  to  decide  whether  or  not  to  accept  the  clearance  and  if  so, 
when  to  start  the  descent.  The  model  was  exercised  for  a  variety  of  scenarios.  The  experimenters  then 
collected  simulator  data  with  four  two-pilot  crews.  The  behavior  of  the  model  was  comparable  to  that  of 
the  human  pilots. 

Analysis  Utilities  and  Interface  Support: 

The  MIDAS  support  environment  has  editors  and  browsers  for  creating  and  changing  system  and 
equipment  specifications,  and  operator  procedures  and  tools  for  viewing  and  analyzing  simulation  results. 
Currently  much  specialized  knowledge  is  required  to  use  these  tools  to  create  models,  but  it  is  worth 
noting  that  a  major  thrust  of  the  MIDAS  redesign  is  to  develop  a  more  self-evident  GUI  that  will  allow 
nonprogrammers  and  users  other  than  MIDAS  development  staff  create  new  simulation  experiments  using 
MIDAS.  In  addition,  this  version  will  eventually  include  libraries  of  models  for  several  of  the  more 
important  domains  of  MIDAS  application  (rotorcraft  and  fixed-wing  commercial  aircraft). 

Availability,  Cost,  and  Contact  Information: 

NASA  AMES 

http://humansystems.arc.nasa.  gov/  groups/midas/ 


Acquisition  Directorate 

Research  &  Development  Center 


B-36 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


APPENDIX  C.  TOOLS  FOR  DETERMINING  MANNING  AND  PERSONNEL 
QUALIFICATIONS 
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C.l  Advisor  3.5 

Description 

A  decision  support  tool  to  help  manage  training  budgets  and  resources.  Helps  identify  ways  to  conduct 
training  programs  more  effectively.  Comprised  of  four  modules:  Where  training  funds  should  be  allocated, 
training  effectiveness,  training  delivery,  money/resources  needed. 

Tool/Method  Content 

Theoretical  Assumptions: 

Seems  somewhat  atheoretical.  Job  and  skill  analysis  is  a  standard  theory-free  procedural  analysis. 

HSI  Domains  Addressed: 

•  Training 

Questions  Addressed: 

•  Determine  most  cost-effective  way  to  deliver  training 

•  Effectiveness/cost  of  alternate  training  methods 

•  Estimate  time  required  to  develop  various  training  materials 

•  Return  on  investment  (ROI)  of  alternate  training  methods 

Content  Modeled: 

Feasibility  and  effectiveness  of  alternative  training  delivery  methods.  Computes  &  compares  costs  of 
alternative  methods. 

Model  Granularity: 

Unknown 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

•  Costs  associated  with  trainees,  instructors,  development,  facilities,  maintenance  and  hardware 

•  Training  strategies 

•  Requirements 

•  ROI 

•  Cost  and  time  required 

•  Optimize  strategies,  resources,  cost,  ROI 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

No  explicit  integration  discussed.  Acquisition  team  could  use  this  system  early  in  acquisition  planning  to 
evaluate  alternatives  emerging  from  CONOPS  definition  with  respect  to  training  costs,  schedules,  etc.  No 
known  integration  with  other  environments. 

Computer  Language  and  Interface  Support: 

Interfaces  provided  with  each  analysis  module  to  assist/guide  analyst  through  the  overall  process. 

Interface  examples,  showing  input  support  and  analysis  alternatives,  are  shown  below. 
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Annual  Cost  Distribution  for  Classroom 

□ 
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100  % 

Delivery  Options  Costs 


Figure  C-l.  Input  and  analysis  screens  from  Advisor  3.5. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Conducts  a  cost  and  ROI  analysis,  given  inputs  of  training  strategies  and  delivery  options. 
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Error  Definitions/Taxonomy: 

Training  requirements  based  on  the  Difficulty,  Importance,  Frequency  (DIF)  model. 

Risk  Computation/Mitigation  Methodology: 

Addressed  by  cost  and  ROI  analysis;  training  budgets  and  resources  analysis.  No  mitigation  methodology 
contained  in  system. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

•  Training  required 

•  Skill  requirements  by  job 

•  Media 

•  Cost 

•  ROI  analysis  results 

•  Resource  requirements 

•  Implementation  requirements 

Development  Steps  Supported: 

•  Training  requirements  analysis 

•  Identify  re-training  intervals 

•  Identify  training  costs,  systems,  ROI 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

System  requirements  map  to  training  requirements  through  the  Advisor  module  1  requirements  analysis. 
The  content  of  this  analysis  should  reflect  system  requirements  arising  from  the  SE  requirements  analysis. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

System  seems  stable  and  has  apparently  be  validated  through  numerous  uses  in  many  domains.  Platform 
requirements  include  Intel  processor,  32  MB  RAM,  20  MB  HD  space,  VGA  or  SVGA. 

Analysis  Utilities  and  Interface  Support: 

Interface  support  provided  through  GUI.  Tech  support  available  through  BNH  Software. 

Availability,  Cost,  and  Contact  Information: 

BNH  Software,  4000  Steinberg  Street 
St.  Laurent,  QC,  Canada  H4R  2G7 

Tel:  (800)747-4010(514)745-4010,  Fax:  (800)947-4010(514)745-4011 
E-mail:  sales@bnhexpertsoft.com 
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C.2  C3TRACE 

Description 

Evaluate  the  effects  of  different  personnel  configurations  and  information  technology  on  human 
performance  as  well  as  on  overall  system  performance. 

Tool/Method  Content 

Theoretical  Assumptions: 

Discrete  event  system  based  on  micro-Saint.  Makes  the  same  theoretical  assumptions. 

HSI  Domains  Addressed: 

•  Personnel 

Questions  Addressed: 

•  Operator  utilization 

•  Task  completion  results 

•  Message  flow  &  success 

•  Utilization  over  time 

•  Operator  performance 

•  Task  summary/timeline 

Content  Modeled: 

Operator,  organizational  performance  and  system  effectiveness  subject  to  operational  conditions  and 
constraints 

Model  Granularity: 

Spans  operator  level  to  organizational  level. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Unknown 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Trade-off  analysis  of  system  alternatives.  Potential  integration  with  environment  and  system  simulations. 
There  are  plans  to  integrate,  at  some  level,  with  large-scale  simulation  environments  and  detailed 
cognitive  modeling  environments 

Computer  Language  and  Interface  Support: 

Proprietary  language  developed  for  IMPRINT.  Interface  for  developing  task  networks,  defining  variables, 
parameterizing  models. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risk  assumptions  embedded  in  parameterization  process. 
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Error  Definitions/Taxonomy: 

No  explicit  definition  of  any  risk  language. 

Risk  Computation/Mitigation  Methodology: 

Risk  computations  are  implicit  in  the  analyses  of  task  networks  and  performance  outcomes.  Risk 
mitigation  is  accomplished  through  alternative  designs  and  modifications  of  task  dynamics 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Task  performance  times,  task  performance  accuracies,  system  performance  outcomes,  errors 

•  System  alternatives 

•  Effectiveness  associated  with  varying  strategies  of  work  configurations 

Development  Steps  Supported: 

•  Requirements  analysis 

•  Analysis  of  alternatives 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Reflected  in  the  task  networks  developed  for  alternative  system  configurations. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

No  formal  reliability  or  validity  work  yet.  Validation  is  planned.  Platform  requirements  include  PC, 
windows,  VGA,  100  MB  hard  drive  space. 

Analysis  Utilities  and  Interface  Support: 

Task  network  definition,  task  parameterization,  debugging 

Availability,  Cost,  and  Contact  Information: 

Patricia  Kilduff 
Army  Research  Lab 
Aberdeen  Proving  Ground,  MD 
(410)  278-5874 
pkilduff@arl.army.mil 

C.3  Job  Assessment  Software  System  (JASS) 

Description 

Computer-based  survey  tool  used  to  identify  and  rate  the  level  of  skills  and  abilities  needed  to  perform  jobs 
and  job  duties.  Based  on  Fleishman’s  taxonomy. 

Tool/Method  Content 

Theoretical  Assumptions: 

Fleishman  taxonomy 


Acquisition  Directorate 

Research  &  Development  Center 


C-6 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


HSI  Domains  Addressed: 

•  Training 

•  Personnel 

•  Manpower 

•  HFE 


Questions  Addressed: 

•  Skill  requirements 

•  Skill  levels 

•  Training  retention  intervals 
Content  Modeled: 

Cognitive,  perceptual,  psycho-motor  skills  and  abilities  required  for  defined  jobs 

Model  Granularity: 

Individual  skills  by  job 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Scale  scores  written  to  an  Access  database,  organized  by  skill.  All  definitions  are  based  on  Fleishman’s 
50  generic  abilities. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Could  be  used  with  requirements  analysis  to  estimate  capacity  for  job  accomplishment  and  training 
requirements  for  defined  jobs 

Computer  Language  and  Interface  Support: 

No  computer  language.  There  is  a  GUI  for  survey  creation  and  implementation  of  the  skills/abilities  index 
for  each  job  (see  Figure  C-2). 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risk  not  addressed  by  this  tool. 

Error  Definitions/Taxonomy: 

Not  addressed  by  this  tool. 

Risk  Computation/Mitigation  Methodology: 

Not  addressed  by  this  tool. 


Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Indexes  required  skills  &  abilities  for  job  assignments  defined  through  system  allocation  activities. 


Development  Steps  Supported: 
Function  analysis  and  allocation 
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Figure  C-2.  JASS  interface  for  identifying  and  rating  skills  and  abilities  needed  to  perform  a  job. 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Relates  requirements  and  allocation  decisions  to  abilities  and  training  requirements. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

No  information. 

Analysis  Utilities  and  Interface  Support: 

•  GUI  for  development  and  implementation  of  surveys 

•  GUI  for  export  into  Access 

•  Output  report  displays  mean  and  SD  of  scores  for  each  skill 

Availability,  Cost,  and  Contact  Information: 

Barry  Tillman 

NASA  Johnson  Space  Center 

(281)483-7131 

barry.tillman-l@nasa.gov 
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C.4  Manpower  Analysis  and  Prediction  System  (MAPS) 

Description 

MAPS  is  a  simulation  model  developed  at  the  Naval  Surface  Warfare  Center,  Carderock.  It  mimics  the 
Navy  Manpower  Requirements  System  (NMRS)  (using  NMRS  inputs  as  a  baseline)  but  is  simpler  to  use 
and  provides  valuable  quick-response  capability.  Unlike  NMRS,  MAPS  contains  a  billet-cost  component, 
making  it  possible  to  predict  costs.  The  costs  in  MAPS  include  all  MPN  and  OMN  cost  of  personnel  - 
salaries,  recruiting,  training,  housing,  and  medical  expenses.  It  also  creates  linkages  between  the  Required 
Operational  Capability/  Projected  Operational  Environment  (ROC/POE)  and  manpower  requirements, 
which  could  prove  useful  to  policymakers. 

MAPS  permits  ship  mission  planners  to  evaluate  billet  and  cost  impacts  before  a  final  ROC  is  approved  and 
published.  Hardware  design  teams  can  research  the  manpower  impact  of  proposals  while  in  the  concept 
phase. 

Tool/Method  Content 

MAPS  is  a  relational  database  based  on  functional  data  from  existing  ship  classes.  Within  MAPS,  this  data 
has  direct  linkage  to  all  applicable  ROC  elements.  Billet  calculations  follow  approved  Navy  standards. 
Various  billet  and  cost  reports  are  available  from  the  system  on  demand.  Because  of  the  relational  database 
configuration,  rapid  response  to  proposed  changes  is  available.  Changes  to  ROC,  on-board  equipment, 
operational  stations,  or  ship  compartments  can  be  quickly  assessed  and  processed.  Selected  functional  areas 
can  be  isolated  for  in  depth  analysis.  When  specific  billets  are  identified  to  the  system,  their  relationship  to 
functions,  task,  and  ROC  elements  can  be  identified  and  evaluated.  Multiple  billet  identities  can  be 
obtained,  usually  in  real  time  and,  if  desired,  complete  with  cost  data. 

The  calculations  performed  by  the  system  adhere  to  approved  Navy  standards  for  variables  such  as 
workweek,  productivity  allowance,  make-ready  /  put  away,  etc. 

HSI  Domains  Addressed: 

This  tool  addresses  the  determination  of  manpower  and  personnel  requirements. 

Questions  Addressed: 

MAPS  can  identify  associated  personnel  changes  associated  with  system  changes. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  original  MAPS  program  was  developed  in  1997-98.  The  latest  version  has  been  rewritten  in  Visual 
Basic  6.0  and  the  data  structures  have  been  modified  to  permit  multiple  baselines  within  the  same 
application.  MAPS  is  based  on  functional  data  from  existing  ship  classes.  Within  MAPS,  these  data  have 
direct  linkages  to  all  applicable  ROC  elements.  Billet  calculations  follow  approved  Navy  standards. 

Various  billet  and  cost  reports  are  available  from  the  system  on  demand. 
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The  following  reports  can  be  generated  from  the  MAPS: 

Summary  Report:  The  Summary  Report  displays  the  manpower  requirements  calculated  including  rating, 
paygrade  and  Navy  Enlisted  Classification  (NEC)  and  summarized  by  Division  and  Department.  Cost 
information  for  the  billets  can  also  displayed  and  summarized.  Cost  data  include  No  Cost  Data,  Total  Cost, 
Direct  Cost  and  Indirect  Cost. 

High  Driver  Report:  The  High  Driver  Report  displays  project  calculated  manpower  requirements  by  rating 
and  Division  summed  by  the  category  of  workload  such  as  Preventive  and  Corrective  Maintenance, 
Facilities  Maintenance,  Own  Unit  Support,  Directed  Requirements,  and  Watch  Stations. 

NEC  Requirements  Report:  The  purpose  of  this  report  is  to  identify  those  NEC’s  that  are  required  within 
the  project,  along  with  the  associated  ratings  and  paygrades. 

Project  Task  Report:  The  Project  Task  Report  displays  all  project  Workload  Factors,  e.g.,  Facilities 
Maintenance  (FM),  Preventive  Maintenance  (PM),  Watch  Stations  (WS),  etc.,  and  all  Task  Identifications 
and  Task  Names  within  those  Workload  Factors. 

Risk  and  Trade-off  Management 

This  has  yet  to  be  identified. 

Traceability 

Development  Steps  Supported: 

•  Performance/workload/training  estimation 

•  Personnel  selection 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Analysis  Utilities  and  Interface  Support: 

MAPS  provides  a  complete  interface  to  support  all  aspects  of  project  creation,  manpower  specifications, 
and  analysis.  The  screenshots  presented  below  provide  examples  of  MAPS  project  set-up  and  analysis 
selection  support. 

Availability,  Cost,  and  Contact  Information: 

Multi-Media  Communications,  Inc 
6610  Rockledge  Drive,  Suite  168 
Bethesda,  Maryland  20817 
(301)  897-8777 
postmaster@mmci.net 

Mr.  Bill  Cheng 
NSWC  Carderock  Division 
West  Bethesda,  Maryland  20817 
(301)  227-1926 
chengh@navsea.navy.mil 
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Figure  C-3.  Screens  from  Navy’s  manpower  analysis  and  prediction  system. 
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C.5  Navy  Manpower  Requirements  System  (NMRS) 

Description 

The  Navy  Manpower  Requirements  System  (NMRS)  is  the  process  the  Navy  uses  to  determine  the  number 
of  sailors  it  needs  on  board  ships.  This  model  computes  billets  from  expected  workload  using  certain 
assumptions  about  hours  of  work,  paygrade,  and  workload  allowances. 

The  NMRS  is  currently  undergoing  a  major  update.  The  Phase  II  system  will  be  a  Coast  Guard  owned  and 
operated  enterprise  level  application  and  will  perhaps  serve  as  a  benchmark  for  the  Navy’s  effort  to  update 
their  MRD  information  system  technologies. 

Tool/Method  Content 

Theoretical  Assumptions: 

•  Navy  Standard  Workweek, 

•  Staffing  Tables  from  the  Navy’s  Standard  Organization  and  Regulations  Manual  (SORM) 

HSI  Domains  Addressed: 

This  tool  addresses  the  determination  of  manpower  and  personnel  requirements. 

Questions  Addressed: 

•  How  many  personnel  (minimum/maximum)  are  required  to  operate  the  system? 

•  What  skills  are  required  of  the  personnel? 

Content  Modeled: 

In  Phase  II,  The  USCG  is  capturing  the  IT  functional  requirements  for  an  MRD  AIS  based  upon  our 
knowledge  and  experience  with  other  DOD  methods/applications  and  practices  and  guidance  from  the 
leading  Industrial  Engineering  Text  (Maynard’s). 

A  concurrent  Analysis  of  Alternatives  (AoA)  will  identify  potential  systems  to  assist  with  developing 
criterion  for  phase  3  (Development).  Phase  three  may  include  Government-off-the-shelf  technology, 
commercial-off-shelf  technology,  a  combination,  or  development  from  scratch. 

Model  Granularity: 

Low.  Units  of  analysis  are  large  work  units  and  people  assigned  to  complete  work  units. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

NAV-MAC  uses  the  NMRS  generate  the  Ship  Manpower  Documents  (SMDs),  which  give  the  number 
and  type  of  billets  needed  to  man  a  particular  ship  class.  The  SMDs  are  used  to  generate  the  Activity 
Manpower  Documents  (AMDs)  which  provide  the  same  information  but  in  more  detail.  They  do  not 
include  recommendations  for  optimizing  personnel  or  performance  objectives. 
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Risk  and  Trade-off  Management 

There  are  a  number  of  trade-offs  associated  with  the  system. 

•  The  algorithm  must  be  run  by  the  Navy  Manpower  Analysis  Center.  The  system  is  not  opaque  and 
the  documentation  is  not  readily  accessible  or  processed. 

•  The  NMRS  billet-generation  algorithm  is  based  on  assumptions  that  are  decades  old.  The  Navy 
Standard  Workweek,  paygrade  tables,  and  workload  allowances  date  from  the  1960s  and  1970s.  As 
a  result,  they  may  be  incompatible  with  today’s  technology,  personnel  policies,  workforce,  and 
business  practices.  The  system,  however,  is  in  the  midst  of  a  large  multi-phase  overhaul  to  make  the 
system  user-friendly  and  perhaps  web-enabled. 

•  The  process  does  not  adequately  consider  manning  alternatives.  In  setting  requirements,  NAVMAC 
takes  technology  as  given  and  uses  decades-old  assumptions  about  average  hours  of  work  and  the 
paygrade  mix  of  the  crew.  For  example,  it  does  not  take  into  account  any  reductions  in  personnel 
gained  by  using  technology. 

Traceability 

In  its  current  form,  the  NMRS  process  is  not  transparent.  The  NMC  works  with  the  user  to  collect  system 

inputs  and  NMC  personnel  use  the  NMRC  to  generate  data  on  the  user’s  behalf. 

Development  Steps  Supported: 

•  Requirements  analysis 

•  Performance/workload/training  estimation 

•  Personnel  selection 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Availability,  Cost,  and  Contact  Information: 

CDR  Troy  S.  Taylor  (Trov.Tavlor@navy.mil ) 

Coast  Guard  Human  Resources  Deepwater  (CG-1B3) 

CGLO  to  Navy  Manpower  Analysis  Center  (NAV-MAC) 

Tel:  901-874-6233;  Fax:  901-874-6448 


Acquisition  Directorate 

Research  &  Development  Center 


C-13 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


This  page  intentionally  left  blank. 


Acquisition  Directorate 

Research  &  Development  Center 


C-14 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


APPENDIX  D.  WORKLOAD  AND  SITUATION  ASSESSMENT  ANALYSIS  AND 
DESIGN  TOOLS 
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D.l  CART/IMPRINT  Pro 


Description 

Discrete  event  task  network  descriptions  of  mission  outcomes,  operator  workload,  taxon  loadings. 

Tool/Method  Content 

Theoretical  Assumptions: 

All  complex  activities  can  be  expressed  as  task  networks.  System  effectiveness  is  a  function  of  task 
network  dynamics.  Human  contribution  to  effectiveness  affected  by  task  goals  and  constraints, 
performance  functions  and  parameters,  and  moderators. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

•  Manpower 

•  Personnel 

Questions  Addressed: 

•  Task  completion  (Y/N) 

•  Task  accuracy 

•  Task  completion  times 

•  VACP  Workload 

•  Roles  of  performance  multipliers 

Content  Modeled: 

•  Goals 

•  Procedures  and  tasks 

•  Behaviors 

•  Workload 

•  Performance  multipliers 

•  Stressors 

•  Task  variability 

Model  Granularity: 

Moderate 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

•  Task  time 

•  Task  accuracy 

•  All  analysis  done  within  the  language  of  task  networks 

•  Results  presented  in  terms  of  times  and  accuracies 

•  Goals,  functions,  tasks 

•  Output  related  to  system  development  cast  in  terms  of  system  effectiveness 

•  All  definitions  are  in  terms  of  task  networks,  discrete  event  processes,  human  performance  factors. 
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Methods  used  to  Integrate  with  SE  and  Other  Environments: 

•  Capture  human  component  of  system  model 

•  Operationalize  CTA 

•  Define  human  participation  in  CONOPS 

•  Communicate  human  component  functionality  to  systems  engineering 

•  Perform  system  effectiveness  trade-offs  with  respect  to  human  performance 

•  Model  CWE  for  traceability 

•  HLA 

•  DIS 

Computer  Language  and  Interface  Support: 

C#  used  in  latest  version  of  MicroSaint  engine  supporting  IMPRINT.  Interface  support  includes: 

•  GUI  support  of  task  network  construction 

•  Form-based  support  for  task  specifications,  workload,  crew  features,  taxons,  decision  paths 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

No  special  assumptions  regarding  risk.  No  adherence  to  any  theoretical  risk  model. 

Error  Definitions/Taxonomy: 

Modeler-defined 

Risk  Computation/Mitigation  Methodology: 

Stochastic  outcomes  based  on  modeler-defined  assumptions  and  structure/specification  of  the  task 
network  under  study. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

•  Behaviors 

•  Time 

•  Accuracy 

•  Relates  human  performance  considerations  to  overall  system  effectiveness. 

Development  Steps  Supported: 

•  Cognitive  work  analysis 

•  System  modeling  &  requirements  analysis 

•  Risk  analysis  and  management 

•  Workflow  concept  development 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Allows  modeler  to  represent  relationships  between  system  requirements  and  HFE,  training,  manpower, 
environment. 


Other  Evaluation  Criteria 

Learning  curve:  Moderate 
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Reliability,  Validity,  and  Platform  Requirements: 

System  is  stable  and  considered  reliable  in  its  output.  Verification  testing  of  each  version  at  Army 
Research  Labs.  Underlying  structure  passed  VV&A  in  1995. 

Platform  requirements  include  Windows  XP  or  2000,  512  MB  RAM,  10  GB  hard  drive  space 
1280x1024  32  bit  color  display,  Windows  Installer  3.1 

Analysis  Utilities  and  Interface  Support: 

The  CART/IMPRINT  Pro/MicroSAINT  family  has  extensive  user  and  interface  support  extending  across 
all  aspects  of  model  development,  specification/parameterization  and  analysis.  Figure  D-l,  which  shows 
part  of  an  analysis  of  an  Army  tank  crew,  addresses  these  human  performance  modeling  stages.  The 
upper  right  screen  shows  a  task  network  in  development.  A  specification  interface  (not  shown)  enables 
modelers  to  specify  model  parameters  (such  as  time  on  task,  workload,  etc.).  The  middle  screen  shows  the 
workload  of  one  crew  member  in  a  tabular  format.  The  final  screen  demonstrates  a  workload  analysis  of 
all  four  crew  members. 

Availability,  Cost,  and  Contact  Information: 

Free  but  limited  to  US  government  agencies,  US  industry  and  universities  with  government  contracts. 

Chameta  Samms 
Army  Research  Lab 
(410) 278-5877 
imprint-info@arl.army.mil 
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Figure  D-l.  IMPRINT  Pro  model  development  and  analysis  screens. 
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D.2  Designer’s  Situation  Awareness  Tool  (DeSAT) 

Description 

The  Designer’s  Situation  Awareness  Toolkit  (DeSAT)  aids  designers  in  creating  systems  that  support 
situation  awareness  (SA).  The  SA-Oriented  design  approach  involves  three  phases:  an  analysis  of  SA 
requirements,  the  application  of  SA-Oriented  design  principles,  and  the  measurement  of  SA  during  design 
evaluation.  DeSAT  provides  support  to  the  designer  for  each  phase  of  the  SA-Oriented  design  process 
through  both  tutorials  and  application  specific  tools.  The  DeSAT  tool  suite  includes  several  tools  that 
support  various  phases  of  design.  These  include: 

(1)  Tools  for  SA  requirements  analysis.  These  tools;  known  as  the  Goal-directed  Task  Analysis 
(GDTA)  tool,  GDTA  checklist  tool  and  GDTA  tutorial;  support  early  design  stages.  The  GDTA 
tool  helps  analysts  create  graphic  representations  of  the  task  analyses  done  in  the  initial  stages  of  a 
design  effort  by  documenting  operator  goals,  decisions  and  situation  awareness  requirements.  The 
checklist  tool  helps  analysts  assess  designs  to  ensure  that  requirements  are  met,  as  well  creating  a 
list  of  requirements  missing  from  the  design. 

(2)  Tools  for  situation  awareness-based  design.  These  include  design  guidelines  tools  consisting  of 
checklists  and  design  principles  and  a  situation  awareness-oriented  design  tutorial  describing  50 
design  principles. 

(3)  Situation  awareness  measurement  tools  based  on  the  SA  Global  Assessment  Technique  (SAGAT). 
SAGAT  contains  support  for  SA  query  creation,  administration  of  the  survey  and  results;  and 
formatting  support  for  export  to  statistical  packages. 

Tool/Method  Content 

Theoretical  Assumptions: 

SA  consists  of  an  awareness  of  what  is  happening  around  you  and  what  implications  information  has  for 
the  future  behavior  of  a  system.  There  are  typically  thought  to  be  3  levels  of  SA:  (1)  perceiving  critical 
factors  in  the  environment,  (2)  understanding  what  those  factors  mean,  particularly  when  integrated 
together  in  relation  to  the  operator’s  goals,  and  (3)  understanding  what  will  happen  with  the  system  in  the 
near  future. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

The  DeSAT  is  designed  to  address  HSI  domains  that  are  sensitive  to  an  awareness  of  information  and 
situational  dynamics,  team  processes,  functional  meaning  with  respect  to  task  requirements  and  decision 
making.  These  include  the  domains  of  human  factors  engineering,  training  and,  potentially,  manpower 
and  personnel  in  a  derivative  manner. 

Questions  Addressed: 

•  Information  structuring,  managing  uncertainty  &  complexity 

•  Decision  support 

•  Training  requirements  and  estimates  of  training  times 

•  Visual  &  Auditory  displays 

•  Team  operations 

•  Automation,  controls,  alarms  and  alerts 

•  Controls 
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Content  Modeled: 

This  method  makes  assumptions  about  task  structure  and  information  dynamics,  and  attempts  to  identify 
and  measure  levels  of  awareness  and  understanding  about  these  attributes.  At  level  1  SA  the  content 
modeled  by  the  DeSAT  system  consists  of  domain  specific  elements  such  as  aircraft  type,  heading, 
altitude  and  flight  plan.  Level  2  content  consists  of  descriptions  of  the  meaning  of  the  level  1  elements, 
with  respect  to  the  goals  that  an  operator  has.  Level  3  content  is  made  up  of  predictions  of  “system” 
behavior  that  an  operator  generates  based  on  the  information  at  the  lower  levels.  Because  the  tool  relies 
on  the  existence  of  a  goal-directed  task  analysis  for  the  identification  and  measurement  of  SA,  goals  also 
must  be  explicitly  modeled. 

Model  Granularity: 

Processes  and  information  can  be  described  to  as  low  a  level  as  desired  by  analysts.  However,  levels  of 
granularity  are  practically  limited  in  that  the  theory  is  descriptive  and  non-quantitative. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  comes  in  the  form  of  design  recommendations  and  specifications  based  on  DeSAT  analysis  during 
the  early  portions  of  design.  Terminology  is  that  of  standard  situation  analysis  research.  There  are  no 
specific  taxonomic  conventions  with  DeSAT,  other  than  those  associated  with  the  three  descriptive  levels 
of  SA  theory. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

There  do  not  seem  to  be  any  formalized  methods,  contained  within  DeSAT,  that  can  be  used  to  integrate 
the  methodology  or  tool  output  with  environments  in  the  systems  engineering  domain.  In  the  absence  of 
such  methods,  it  is  left  up  to  the  SA-oriented  analyst  to  integrate  DeSAT  outputs  into  the  overall  system 
design  process. 

Computer  Language  and  Interface  Support: 

The  system  requires  installation  of  the  MySQL  database  and  WxPython  GUI  programming  toolkit. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

There  are  no  specific  facilities  for  defining  or  analyzing  errors  in  DeSAT.  Some  attention  to  error  is 
contained  in  the  design  principles  contained  in  the  design  guidelines  tool.  These,  however,  are  general 
rules  of  thumb  for  design  rather  than  focused  error  or  risk  analysis  tools. 

Error  Definitions/Taxonomy: 

No  specific  error  definitions  or  taxonomy. 

Risk  Computation/Mitigation  Methodology: 

None. 

Traceability 

The  DeSAT  system  provides  facilities  that  allow  assessment  of  aspects  of  designs  that  satisfy 
requirements,  as  well  as  a  method  of  documenting  what  requirements  are  missing  from  a  design.  This  is 
done  through  a  checklist  mechanism  contained  in  the  GDTA  checklist  tool.  Other  than  this  checklist 
enumeration  there  seems  to  be  no  formal  mechanism  for  design  traceability.  Furthermore,  the  satisfaction 
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of  requirements  provided  by  the  GDTA  checklist  tool  seems  to  address  only  those  “requirements”  that  are 
specific  to  SA.  This  limits  the  utility  of  the  tool  with  respect  to  broader  system  requirements. 

The  DeSAT  supports  several  development  steps,  although  only  partially.  These  include  requirements 
analysis;  function  analysis  and  allocation;  task  design;  interface  and  team  development;  and  performance, 
workload  and  training  estimation.  The  tool  also  can  contribute  to  training  development  and  performance 
assurance. 

There  are  no  explicit  methods  for  mapping  system  requirements  to  HSI  requirements.  The  DeSAT  does 
not  address  any  HSI  domain  other  than  HFE  and  training. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

The  SAGAT  portion  of  DeSAT  has  been  validated  across  a  wide  variety  of  analytical  and  design  efforts. 
DeSAT  requires  450  MB  of  memory  for  a  full  installation.  The  system  runs  on  Windows  platforms. 
MySQL  and  WxPython  are  required. 

Analysis  Utilities  and  Interface  Support: 

A  comprehensive  user  manual  is  included  with  the  system  along  with  tutorial  information  for  the  specific 
modules.  There  is  no  mention  of  online  or  technical  support  on  the  tool  website.  Interface  support  is 
provided  by  GUI  facilities  for  each  of  the  tool  components. 

Availability,  Cost,  and  Contact  Information: 

Cost  depends  on  the  number  of  copies  purchased  and  ranges  from  $1400  for  5  or  more  individual  copies 
to  $20,000  for  a  site  license.  Contact  information  is  shown  below: 
http://www.satechnologies.com/products/ 

D.3  Cognitive  Reliability  and  Error  Analysis  Method  (CREAM) 

Description 

A  comprehensive  approach  to  Human  Reliability  Analysis  (HRA)  that  includes  a  method  to  conduct  an 
analysis  that  can  be  used  to  both  search  for  the  causes  of  errors  and  predict  performance,  an  error 
classification  scheme  that  consists  of  a  number  of  groups  that  describe  person-related,  technology-related, 
and  organization-related  errors,  and  an  underlying  model  of  operator  cognition  called  COCOM  (Contextual 
Control  Model)  that  describes  how  actions  are  chosen  based  on  the  result  of  the  interaction  between 
competence  and  context. 

Tool/Method  Content 

Theoretical  Assumptions: 

Human  work  can  be  characterized  by  a  scale  of  “doing”  to  “thinking.”  Some  tasks,  such  as  manual  skills 
and  following  a  procedure,  require  much  “doing”  and  little  “thinking,”  while  others,  such  as  diagnosis, 
planning,  and  problem  solving,  require  much  “thinking”  and  little  “doing.”  The  development  of  modem 
technology  has  changed  the  nature  of  human  work  from  being  mostly  manual  skills  to  being  mostly 
knowledge  intensive  functions  (i.e.,  cognitive  tasks).  In  present-day  industrial  environments  the  amount  of 


Acquisition  Directorate 

Research  &  Development  Center 


D-8 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


“thinking”  is  increased  while  the  amount  of  “  doing”  is  reduced.  This  state  of  affairs  has  consequences  for 
both  system  design  and  reliability  analysis.  In  system  design,  for  instance,  conventional  ergonomic  aspects 
must  be  replaced  by  cognitive  ergonomics.  Similarly,  in  risk  assessment  and  reliability  analysis,  first 
generation  HRA  must  be  replaced  by  a  second  generation,  context-dependent  cognitive  reliability 
analysis. 

HSI  Domains  Addressed: 

This  tool  addresses  the  HFE  and  safety  domains. 

Questions  Addressed: 

1 .  Identify  those  parts  of  the  work,  as  tasks  or  actions,  that  require  or  depend  on  human  cognition,  and 
which  therefore  may  be  affected  by  variations  in  cognitive  reliability, 

2.  Determine  the  conditions  under  which  the  reliability  of  cognition  may  be  reduced,  and  where  therefore 
these  tasks  or  actions  may  constitute  a  source  of  risk, 

3.  Provide  an  appraisal  of  the  consequences  of  human  performance  on  system  safety  which  can  be  used 
in  a  PRA/PSA,  and 

4.  Develop  and  specify  modifications  that  improve  these  conditions,  hence  serve  to  increase  the 
reliability  of  cognition  and  reduce  the  risk. 

Content  Modeled: 

Errors,  error  probabilities,  probabilities  of  system  failures  and  relationships  between  these. 

Model  Granularity: 

Variable.  Errors,  in  particular,  can  be  modeled  at  whatever  level  of  granularity  desired  by  analysts. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  primary  form  of  output  from  a  CREAM  analysis  is  a  qualitative  and-or  quantitative  description  of 
errors,  causes  and  the  relationships  between  the  two.  Error  mitigation  strategies  normally  take  the  form  of 
recommendations  for  design  changes  within  the  context  of  the  cognitive  model  underlying  this  method 
(Hollnagel,  E.  (1997).  Context,  cognition  and  control.  In  Y.  Waem,  (Ed.),  Cooperation  in  process 
management:  Cognition  and  information  technology.  London:  Taylor  and  Francis.).  See  the  section  on 
Risk  and  Trade-off  Management  below  for  terminology  and  taxonomic  conventions. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

No  specific  methods  are  described  for  integrating  this  information  with  other  system  engineering 
processes  or  environments.  However,  this  methodology  is  sufficiently  robust  that  taking  input  and 
providing  output  for  other  system  engineering  activities  should  be  straightforward.  Specific  integration 
methods  to  be  used  should  be  developed  by  the  individual  development  team,  tailored  to  the  needs  of  each 
project. 

Computer  Language  and  Interface  Support: 

No  specific  computer  language  or  interface  support  has  been  developed  for  this  method.  However,  see  the 
reference  to  an  early  CREAM  analysis  support  system  in  the  “Analysis  Utilities  and  Interface  Support” 
section  below. 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

This  approach  assumes  three  major  categories  of  causes  that  can  lead  to  a  loss  of  human-machine 
reliability,  each  category  containing  several  specific  manifestations  as  shown  below: 

•  Person-related  causes 

•  Causes  arising  from  specific  cognitive  functions 

•  Causes  based  on  temporary  operator  states 

•  Causes  based  on  permanent  states  of  the  operator 

•  Technology-related  causes 

•  Equipment  malfunctions 

•  Causes  based  on  procedures 

•  Causes  based  on  temporary  interactions  between  human  and  machine 

•  Causes  based  on  permanent  interactions  between  human  and  machine  (design  flaws) 

•  Organization-related  causes 

•  Communication 

•  Organization 

•  Training 

•  Ambient  conditions 

•  Working  conditions 

Error  Definitions/Taxonomy: 

Similar  to  causes,  errors  are  organized  into  a  quasi-hierarchical  set  of  categories: 

•  Actions  taken  at  the  wrong  time 

•  Timing  errors 

•  Duration  errors 

•  Actions  of  the  wrong  type 

•  Force  errors 

•  Distance/magnitude  errors 

•  Speed  errors 

•  Direction  errors 

•  Actions  directed  at  the  wrong  object 

•  Neighbor  errors 

•  Similar  object  errors 

•  Unrelated  object  errors 

•  Actions  in  the  wrong  place 

•  Omission  errors 

•  Jumping  forward  errors 

•  Jumping  backward  errors 

•  Repetition  errors 

•  Reversal  errors 
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Risk  Computation/Mitigation  Methodology: 

Risk  mitigation  strategies,  by  and  large,  consist  of  identifying  error  types,  causes  and  probabilities  that 
causes  of  each  type  will  lead  to  errors  of  each  type.  This  is  carried  out  through  some  combination  of 
retrospective  analysis,  qualitative  and  quantitative  performance  prediction.  Designers  then  eliminate  the 
problems  through  design  changes. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

CREAM  outputs  errors,  error  probabilities  and  system  failure  probabilities  associated  with  various  error 
categories.  These  should  be  used  to  inform  and  constrain  the  specification  of  acquisition  requirements.  In 
cases  where  CREAM  analysis  indicates  catastrophic  system  events,  the  associated  acquisition 
requirements  should  be  re-written  to  mitigate  the  events. 

Development  Steps  Supported: 

Function  analysis 
Function  allocation 
Task  design 

Interface  and  team  development 
Performance/workload/training  estimation 
Requirements  review 
Training  development 
Performance  assurance 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

The  error  analysis  provided  by  CREAM  will  produce  data  that  can  be  mapped  to  system  requirements  in 
the  form  of  constraints  on  system  development,  taking  account  of  human  reliability  and  error/safety 
information  in  the  evolving  system. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

No  systematic  validity  or  reliability  studies  have  been  carried  out,  to  date,  for  this  tool.  Most  of  the 
current  work  addressing  the  CREAM  methodology  is  concerned  with  (1)  applications  of  CREAM  to 
issues  of  human  reliability  for  specific  systems  or  (2)  research  aimed  at  extending  the  theory  and-or 
classification  system  of  the  tool.  Studies  of  tool  validity  and-or  reliability  likely  will  be  delayed  by  some 
time. 

The  CREAM  tool  is  a  methodology  and,  as  such,  has  no  platform  requirements  associated  with  it.  The 
only  existing  software  instantiation  of  CREAM  is  platform-independent. 

Analysis  Utilities  and  Interface  Support: 

Analysis  utilities  are  provided  within  the  CREAM  methodology,  as  outlined  in  Hollnagel  (1998).  There  is 
limited  interface  support  at  this  time.  A  prototype  software  support  system  has  been  developed  at  the 
University  of  Illinois.  This  should,  however,  be  considered  early  beta-level  support.  See 
http://www.ews.uiuc.edu/~serwv/cream/v0.6beta/  for  the  current  version  of  the  CREAM  Navigator. 
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Availability,  Cost,  and  Contact  Information: 

Hollnagel,  E.  Cognitive  Reliability  and  Error  Analysis  Method.  NY:  Elsevier  Science  Inc.,  1998. 

D.4  Technique  for  Human  Error  Rate  Prediction  (THERP) 

Description 

THERP  is  a  technique  for  human  error  rate  prediction  based  on  probabilistic  risk  analysis  and  fault  tree  task 
decomposition  methods.  It  will  allow  analysts  to  predict  human  error  probabilities  and  to  evaluate  the 
degradation  of  human-computer  systems  likely  to  be  caused  by  human  errors  alone  or  in  connection  with 
equipment  malfunctioning,  operational  procedures  or  other  system  and  human  characteristics  that  influence 
complex  system  behavior. 

Tool/Method  Content 

Theoretical  Assumptions: 

This  method  assumes  that  the  success  or  failure  of  operator  actions  are  equivalent  to  that  of  a  system  or 
subsystem.  Thus,  operator  reliability  can  be  assessed  in  the  same  way  as  the  reliability  of  non-human 
components.  This  is  done  by  (1)  identifying  system  functions  that  can  be  influenced  by  human  errors,  (2) 
decomposing  human  tasks  through  detailed  task  analysis,  (3)  estimating  error  probabilities  for  each  task 
using  a  combination  of  expert  judgment  and  data  available  on  the  effects  of  causal  and  moderating  factors, 
(4)  estimating  the  effects  of  the  errors  compiled  in  the  task  analysis  on  system  failure. 

HSI  Domains  Addressed: 

THERP  is  primarily  focused  on  the  safety  and  HFE  domains. 

Questions  Addressed: 

Two  major  categories  addressed  by  this  tool  are  (1)  the  kinds  and  probabilities  of  errors  associated  with 
tasks  that  will  be  carried  out  by  human  operators  and  (2)  the  probabilities  that  the  errors  identified  will 
lead  to  system  failures,  what  the  nature  of  the  failures  will  be,  and  what  mitigating  strategies  can  be 
developed  to  prevent  or  recover  from  the  errors. 

Content  Modeled: 

THERP  allows  analysts  and  designers  to  consider  tasks,  error  categories  associated  with  tasks, 
probabilities  of  error  occurrence  for  each  task,  and  probabilities  that  errors  in  each  category  will  lead  to 
system  failures. 

Model  Granularity: 

Granularity  will  be  determined  by  the  analyst  through  the  task  decomposition,  error  definition  and  the 
manner  in  which  errors  are  related  to  system  failures.  The  tool  situates  analysis  at  the  level  of 
probabilities. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  is  in  the  form  of  probabilities  of  error  and  probabilities  of  system  failures  resulting  from  errors. 
Terminology  and  taxonomic  conventions  are  largely  the  choice  of  the  analyst,  although  much  of  the  error 
taxonomy  often  is  derived  from  Reason’s  (1990)  error  classification. 
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Methods  used  to  Integrate  with  SE  and  Other  Environments: 

The  primary  method  of  integration  is  the  existence  of  system  failure  probabilities  based  on  the  error 
analysis.  There  are  no  integration  methods  inherent  in  the  tool  itself.  The  output  of  a  THERP  analysis  can 
serve  as  input  to  systems  engineering  tools  in  the  safety  and  workstation  design  domains. 

Computer  Language  and  Interface  Support: 

Not  applicable. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risk  is  defined  as  a  function  of  the  probabilities  of  error  occurrence  and  system  failures  as  a  result  of  the 
occurrence  of  errors.  Risk  will  be  expressed  probabilistically.  These  often  will  be  used  to  support  a 
hazard  analysis. 

Error  Definitions/Taxonomy: 

Any  error  taxonomy  can  be  defined  for  use  in  a  THERP  analysis.  Common  taxonomies  include  Reason 
(1990),  Hollnagel  (1998)  and  Endsley  (1999).  Error  taxonomies,  and  relationships  between  errors, 
typically  are  organized  into  tabular  and-or  “fault  tree”  formats.  Examples  of  these  formats,  for  small 
fragments  of  an  analysis,  are  shown  below. 


Table  D-l.  THERP  error  probability  estimation. 


Item 

Checking  Operation 

Human  Error 
Probability 

Error 

Factor 

1 

Checking  routing  tasks  using  written  manuals 

0.1 

5 

2 

Same  as  above  but  without  manual 

0.2 

5 

3 

Special  short-term,  one-of-a-kind  checking  with  alerting 

0.05 

5 
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Risk  Computation/Mitigation  Methodology: 

The  THERP  methodology  involves  five  steps: 

1.  Define  the  system  or  process.  This  involves  describing  the  system  goals  and  functions  and  the 
consequences  of  not  achieving  them.  It  also  requires  identifying  mission,  personnel,  and 
hardware/ software  characteristics . 

2.  Identify  and  list  all  the  human  operations  performed  and  their  relationships  to  the  system  or  process 
tasks  and  functions.  This  requires  an  analysis  of  all  operator  and  maintainer  tasks. 

3.  Predict  error  rates  for  each  human  operation  or  group  of  operations.  Errors  likely  to  be  made  in  each 
task  or  subtask  must  be  identified.  Errors  that  are  not  significant  in  terms  of  system  operation  are 
ignored.  This  step  includes  estimating  the  likelihood  of  each  error  occurring  and  the  likelihood  of  an 
error  not  being  detected. 

4.  Determine  the  effect  of  human  errors  on  the  system  or  process,  including  the  consequences  of  the 
error  not  being  detected.  This  requires  the  development  of  event  trees.  The  left  limbs  of  the  event 
trees  are  success  paths;  the  right  limbs  are  failure  paths.  Probabilities  are  assigned  to  each  path.  The 
tree  reflects  the  effects  of  task  dependence.  The  relative  effects  of  performance-shaping  factors,  e.g., 
stress  and  experience,  are  estimated. 

5.  Develop  and  recommend  changes  that  will  reduce  the  system  or  process  failure  rate.  The 
recommended  changes  can  be  developed  using  sensitivity  analyses,  in  which  factors  and  values  are 
varied  and  effects  monitored.  THERP  makes  no  assumptions  about  the  dependence  or  independence 
of  personnel  behaviors.  Data  are  taken  from  available  sources. 

A  key  aspect  of  THERP  is  the  determination  of  the  probability  that  an  error  or  class  of  errors  will  result  in 
a  system  or  process  failure.  This  probability  is  assigned  a  value  F,.  Branching  trees  are  constructed  to 
determine  the  paths  to  success  and  failure.  The  probability  that  an  error  will  occur  is  given  by  P,.  FPj  is 
the  joint  probability  that  an  error  will  occur  and  that  the  error  will  lead  to  system  failure.  1-  F;Pi  is  the 
probability  that  an  operation  will  be  performed  that  does  not  lead  to  system  failure.  The  probability  that  a 
class  of  errors  will  lead  to  system  failure  is  given  by: 

Qi  =  l-(l-FiP,f 


Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Output  categories  include  error  probabilities  for  tasks  of  interest,  probabilities  of  system  failures,  joint 
probabilities  that  errors  for  selected  tasks  will  cause  certain  system  failures.  This  information  will  be 
useful  in  developing  system  requirements  and  in  ORD  development. 

Development  Steps  Supported: 

•  Function  analysis 

•  Task  design 

•  Performance  assurance 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

The  error  analysis  carried  out  in  THERP  will  contribute  to  relating  overall  system  requirements  and  task 
organization  to  safety  concerns  involving  human  operators. 
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Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

THERP  is  one  of  the  most  mature  tools  available  for  human  reliability  analysis.  As  such,  it  has  been 
thoroughly  validated  across  many  operational  domains.  Its  reliability  has  been  demonstrated  to  be  high. 

There  are  no  specific  platform  requirements  for  the  use  of  THERP.  Since  the  tool  is  an  analytical  method, 
rather  than  a  specific  software  tool,  the  only  platform  requirements  include  a  tool  that  will  support  task 
decomposition  and  a  statistical  analysis  tool  that  will  support  estimation  of  error  probabilities  and  system 
failures. 

Analysis  Utilities  and  Interface  Support: 

None. 

Availability,  Cost,  and  Contact  Information: 

References  on  THERP  methodology  are  widely  available: 

Endsley,  M.R.  (1999).  Situation  Awareness  and  Human  Error:  Designing  to  Support  Human 
Performance.  Proceedings  of  the  High  Consequence  Systems  Surety  Conference,  Albuquerque. 

Hollnagel,  E.  (1998).  Cognitive  Reliability  and  Error  Analysis  Method.  Oxford,  UK:  Elsevier. 

LaSala,  K.P.,  RAC  Publication.  A  Practical  Guide  to  Developing  Reliable  Human-  Machine  Systems 
and  Processes,  January  2002. 

Reason,  J.T.  (1990).  Human  Error.  New  York:  Cambridge  University  Press. 

Swain,  A.D.,  “THERP,”  SC-R-64-1338,  Sandia  National  Laboratories,  Albuquerque,  NM,  August 
1964. 
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APPENDIX  E.  WORKSTATION  AND  COCKPIT  DESIGN  TOOLS 
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E.l  3D  Static  Strength  Prediction  Program  ™  (3DSSPP)  Version  5.0 

Description 

This  is  a  job  design/evaluation  tool  that  can  provide  a  wide  variety  of  information  and  analyses  ranging 
from  predicted  low  back  compression  forces  to  population  strength  capability  information. 

Tool/Method  Content 

Theoretical  Assumptions: 

3DSSPP  is  most  useful  in  the  analysis  of  “slow”  movements  used  in  manual  material  handling  tasks  since 
the  biomechanical  computations  assume  that  the  effects  of  acceleration  and  momentum  are  negligible. 

HSI  Domains  Addressed: 

This  tool  addresses  the  safety  domain;  specifically  Workstation  Design,  Physical  Ergonomics,  Manual 
Material  Handling,  and  Task  Analysis. 

Questions  Addressed: 

•  How  much  compression  does  the  back  feel? 

•  Where  do  the  maximum  forces  occur  during  the  task? 

•  What  percentage  of  people  are  capable  of  performing  this  task? 

Content  Modeled: 

•  Tasks  and  task  sequences 

Model  Granularity: 

It  is  easy  to  enter  coarse  data.  However,  it  is  also  possible  to  provide  more  detailed  inputs  (see  below). 
The  generated  reports  provide  quite  a  bit  of  detail. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

3DSSPP  presents  a  number  of  reports  including: 

•  Task  Input  Summary  that  provides  information  regarding  body  segment  angles,  hand  locations,  and 
hand  force  magnitude  and  direction, 

•  Analysis  Summary  which  summarizes  hand  forces,  L5/S1  disc  compression,  percent  capable,  balance, 
and  coefficient  of  friction, 

•  Reports  of  Anthropometry,  Posture,  Joint  Locations  and  Moments,  and  Strength  Capabilities. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

This  software  integrates  with  the  Energy  Expenditure  Prediction  Program  (EEPP). 

Computer  Language  and  Interface  Support: 

Computer  language  is  not  an  issue  with  these  proprietary  and  self-contained  tools.  The  tool  set  contains  a 
full  range  of  interfaces  for  data  entry,  analysis  and  reporting. 
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Figure  E-l.  Screens  from  3D  Static  Strength  Prediction  Program. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

This  tool  is  to  be  used  in  conjunction  with  other  analysis  methods  and  should  not  be  use  alone  to 
determine  risk.  Analysis  summaries  can  be  provided  in  either  graphical  format  (as  shown  in  bottom  right 
screen  in  Figure  E-l)  or  tabular  format. 

Traceability 

The  3DSSPP  identifies  and  quantifies  the  physical  limits  of  a  design. 
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Development  Steps  Supported: 

•  Function  analysis 

•  Task  design 

•  Performance/workload/training  estimation 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

Platform  requirements  include  Windows  2000  or  XP  with  a  minimum  128  MB  RAM  and  20  MB  hard  disk 
space. 

Availability,  Cost,  and  Contact  Information: 

The  cost  for  a  single  3DSSPP  license  is  $1,495.  Combined  with  the  EEPP,  a  single  license  is  $1,900. 
There  are  discount  for  larger  quantities. 

Distributed  by: 

University  of  Michigan  Office  of  Technology  Transfer 
Center  for  Ergonomics 
University  of  Michigan 

1205  Beal  Avenue,  Ann  Arbor,  MI  48109-2117 

E.2  Energy  Expenditure  Prediction  Program™  (EEPP) 

Description 

This  tool  predicts  metabolic  energy  expenditure  rates  by  summing  up  the  energy  requirements  of  small, 
well-defined  work  tasks  that  comprise  the  entire  job.  It  identifies  specific  work  tasks  that  contribute  heavily 
to  an  overall  high  job  energy  expenditure  rate,  which  facilitates  job  redesign  activities.  The  EEPP  is  useful 
in  designing  new  jobs,  comparing  one  job  to  another,  and  improving  an  existing  job  by  identifying  the 
particular  tasks  that  require  excess  energy  expenditure. 

Tool/Method  Content 

Theoretical  Assumptions: 

This  tool  assumes  that  a  job  can  be  divided  into  simple  tasks  and  that  the  average  metabolic  energy  rate  of 
the  job  can  be  predicted  by  knowing  the  energy  expenditure  of  the  simple  tasks  and  the  time  duration  of 
the  task. 

HSI  Domains  Addressed: 

•  Safety 

•  Habitability 

•  Manpower 

Questions  Addressed: 

•  Is  the  job/task  acceptable? 

•  How  can  this  job  be  redesigned? 
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Content  Modeled: 

•  Tasks  and  task  sequences.  This  information  is  entered  into  tabular  formats  provided  by  the  system,  as 
shown  in  Figure  E-2.  Detailed  information  for  both  tasks  and  task  elements  can  be  provided. 


Model  Granularity: 

The  model  relies  heavily  on  the  quality  of  the  task  breakdown  and  input  in  to  the  analysis.  The  analysis  is 
specific  to  the  gender  and  body  weight  inputs  to  the  model. 


Figure  E-2.  Analyzing  a  manual  task  design  with  the  Energy  Expenditure  Prediction  Program, 
(from  www.engin.umich.edu/dept/ioe/ENGEXP/) 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Graphical  outputs  suitable  for  reports  are  standard  in  this  software  package. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

This  software  integrates  with  the  3D  Static  Strength  Prediction  Program 
Computer  Language  and  Interface  Support: 

Computer  language  is  not  an  issue  with  these  proprietary  and  self-contained  tools.  The  tool  set  contains  a 
full  range  of  interfaces  for  data  entry,  analysis  and  reporting. 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

This  tool  is  to  be  used  in  conjunction  with  other  analysis  methods  and  should  not  be  use  alone  to 
determine  risk. 

Traceability 

The  EEPP  identifies  and  quantifies  the  physical  limits  of  a  design. 

Development  Steps  Supported: 

•  Function  analysis 

•  Task  design 

•  Performance/workload/training  estimation 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

Platform  requirements  include  Windows  2000  or  XP  with  a  minimum  32  MB  RAM  and  1  MB  hard  disk 
space. 

Availability,  Cost,  and  Contact  Information: 

The  cost  for  a  single  EEPP  license  is  $695.  Combined  with  the  3DSSPP,  a  single  license  is  $1,900.  There 
are  discount  for  larger  quantities. 

Distributed  by: 

University  of  Michigan  Office  of  Technology  Transfer 
Center  for  Ergonomics 
University  of  Michigan 

1205  Beal  Avenue,  Ann  Arbor,  MI  48109-2117 


E.3  LOCATE 

Description 

Computer-aided  tool  for  analyzing  communication  in  visual,  auditory,  tactile  and  movement  domains  in 
multi-operator  machine  workspace  layout  problems.  Computes  link  strength  for  human-human,  human- 
machine  and  machine-machine  combinations.  Rolls  these  up  into  a  single  cost  function.  Can  be  used  to 
form  matrices  of  component  costs. 

Tool/Method  Content 

Theoretical  Assumptions: 

None 

HSI  Domains  Addressed: 

•  HFE 


Questions  Addressed: 

Human-human,  human-machine,  machine-machine  interface  costs  and  effectiveness. 
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Content  Modeled: 

Communication  paths  between  system  components. 

Model  Granularity: 

Individual  human  and  machine  components. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Layouts  of  multi-operator-machine  workspaces.  Can  also  produce  panel  design  layouts.  Output 
characterized  in  terms  of  components,  links,  communication  channels,  positions,  angles. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Addresses  workstation  layout  design,  allocation,  and  system  effectiveness. 

Computer  Language  and  Interface  Support: 

Uses  a  proprietary  language.  Contains  a  GUI  for  entering  all  data  and  conducting  analyses.  Information 
is  entered  via  a  tool  palette  and  work  area.  Following  entry  of  specifications  the  work  environment  can  be 
either  manipulated  manually  by  the  analyst  to  evaluate  workstation  layouts  and  communication 
efficiencies  or  analysts  can  choose  to  allow  the  system  to  optimize  these. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risk  increases  as  an  inverse  function  of  the  cost  score  computed  by  LOCATE’s  AIM  algorithm. 

Error  Definitions/Taxonomy: 

Described  solely  in  terms  of  cost  functions  associated  with  communication  in  various  workspace  layouts 
Risk  Computation/Mitigation  Methodology: 

Uses  optimization  and  cost  function  for  risk  computation.  Mitigations  expressed  as  reductions  in  cost. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

Single  value  of  the  cost  associated  with  a  particular  configuration  and  incremental  costs  associated  with 
each  pairwise  relationship  between  workstations  in  each  communication  domain.  Seems  to  provide  a 
good  way  to  evaluate  workstation  layout  alternatives. 

Development  Steps  Supported: 

Allocation,  workspace  layout,  manpower  estimates,  workload  estimates 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

Can  inform  design  team  about  habitability,  workload,  system  effectiveness  as  a  function  of  workspace 
layout 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 
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Reliability,  Validity,  and  Platform  Requirements: 

Has  been  validity  tested,  system  is  stable.  Platform  requirements  include  Windows  or  Mac  OS-X. 

Analysis  Utilities  and  Interface  Support: 

•  Has  complete  interface  support  through  GUI 

•  Analysis  utilities  offered  through  system  GUI  and  AIM  algorithm 

Availability,  Cost,  and  Contact  Information: 
http://www.sosproducts.ca/LocateVideoOT.html 

E.4  Liberty  Mutual  Tables 

Description 

These  are  loss  prevention  tables  that  provide  guidance  on  the  percentage  of  the  male  and  female  population 
able  to  safely  complete  a  manual  material  handling  task.  It  recommends  that  tasks  be  able  to  be  performed 
by  75%  of  the  female  population. 

Theoretical  Assumptions: 

The  tables  focus  on  lifting  aspects  that  contribute  to  a  high  risk  of  low  back  injury.  The  data  is  based  on 
psychophysical  methodologies  that  include  measuring  oxygen  consumption,  heart  rate,  and 
anthropometric  characteristics. 

HSI  Domains  Addressed: 

This  tool  addresses  the  safety  domain.  More  specifically,  Workstation  Design,  Physical  Ergonomics,  and 
Manual  Material  Handling. 

Questions  Addressed: 

•  What  population  can  safely  complete  this  lift/push/pull/carry? 

•  How  can  the  load  or  type  of  handling  be  modified  to  make  it  safe? 

Content  Modeled: 

•  Object  weight 

•  Hand  distance 

•  Initial/final  hand  height 

Model  Granularity: 

There  are  tables  for  20  different  handling  tasks  (see  example  below).  They  are  gender/load/parameter 
specific.  For  a  greater  numbers  of  input  variables,  and  therefore  greater  detail,  use  the  NIOSH  lift 
equation  instead.  But  since  the  NIOSH  equation  has  many  limitations  on  its  application,  the  Liberty 
Mutual  tables  provide  adequate  detail. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  output  of  these  lookup  tables  is  the  percentage  of  a  gender-specific  population  that  can  perform  the 
task. 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Effective  use  of  these  tables  requires  basic  level  training  in  ergonomics  and  manual  handling  task  analysis 
and  evaluation.  Users  should  be  knowledgeable  of  biomechanical,  physiological  and  psychophysical 
workload  criteria  and  evaluation  methods. 

Traceability 

The  analysis  can  be  conducted  before  and  after  an  intervention  to  demonstrate  that  the  intervention  has 
worked  to  accommodate  a  larger  percentage  of  the  population. 

Development  Steps  Supported: 

•  Function  analysis 

•  Task  design 

•  Performance/workload/training  estimation 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Availability,  Cost,  and  Contact  Information: 

Snook,  SH  and  Ciriello,  VM.  “The  design  of  manual  handling  tasks:  revised  tables  of  maximum 
acceptable  weights  and  forces.”  Ergonomics.  34:9  1197-1213,  1991. 

http://libertymmhtables.libertymutual.com/CM  LMTablesWeb/pdf/LibertyMutualTables.pdf 

Liberty  Mutual  (2004).  Manual  Materials  Handling  Guidelines. 
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Table  E-l.  Liberty  Mutual  table:  female  population  percentages  for  lifting  tasks  ending  above  shoulder 
height  (>53”).  (Source:  From  Liberty  Mutual  (2004).) 
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E.5  Informal  System  Evaluation  Methods 

Descriptions 

Ethnographic  Observation: 

These  are  techniques  that  stem  from  the  anthropology  and  psychology  communities  that  study  how  people 
interact  with  technology.  Domain  practitioners  are  observed  and  interviewed  in  their  actual  work 
environments  as  they  perform  regular  work  activities.  There  are  also  a  host  of  “rapid  ethnography” 
methods  being  developed  by  the  HCI  community  with  the  goal  of  providing  a  reasonable  understanding  of 
workers  and  their  activities  given  significant  time  pressure  and  limited  time  in  the  field.  These  methods 
can  be  useful  in  gathering  user  requirements,  understanding  and  developing  user  models,  and  evaluating 
new  systems  and  iterating  their  design. 

Heuristic  Evaluation: 

A  technique  for  assessing  the  usability  of  a  computer  interface  that  uses  ten  rules  of  thumb,  such  as  “speak 
the  user’s  language,”  “provide  feedback,”  “be  consistent,”  and  “provide  good  error  messages.”  In  a 
heuristic  evaluation,  the  analyst  evaluates  how  well  the  proposed  interface  follows  the  rules  of  thumb  and 
provides  feedback  as  to  how  it  could  be  improved. 

Cognitive  Walk-through: 

In  walk-throughs  and  talk-throughs,  workers  who  know  a  system  perform  a  task  using  an  actual  system  or 
a  realistic  mock-up  for  analysis.  When  performing  a  talk-through,  the  user  is  removed  from  realistic 
surroundings  and  merely  verbalizes  the  demonstration.  Cognitive  walk-throughs  attempt  to  evaluate  the 
state  of  the  user’s  thought  processes  at  each  step  of  task  performance,  with  emphasis  on  identifying 
aspects  of  the  interface  that  are  confusing. 

Interface  Evaluation  Surveys: 

These  are  a  group  of  information  collection  methods  which  can  be  used  to  identify  specific  ergonomics 
problems  or  deficiencies  in  interfaces.  They  address  issues  such  as  the  labeling  and  consistency  of 
controls,  how  well  the  system  works  within  its  environment  (e.g.  is  the  environment  too  noisy  for  an 
auditory  interface?),  and  whether  operators  have  modified  the  system  in  some  way  to  overcome  a 
deficiency  (e.g.  are  there  post-it  notes  everywhere?).  They  are  applied  when  a  detailed  design  has  been 
created.  There  are  also  a  host  of  surveys,  such  as  the  Questionnaire  for  User  Interaction  Satisfaction 
(QUIS),  which  can  be  used  to  assess  worker  satisfaction  with  specific  aspects  of  a  human-computer 
interface. 

Ergonomics  Checklists: 

Checklists  that  an  analyst  can  use  to  ascertain  whether  particular  ergonomic  criteria  are  being  met  by  a 
system.  The  items  within  these  checklists  can  range  from  overall  subjective  opinions  to  very  specific 
objective  checks.  They  can  be  used  to  evaluate  both  existing  and  proposed  systems. 

Tool/Method  Content 

Theoretical  Assumptions: 

All  of  the  informal  system  evaluation  methods  are  essentially  atheoretical  in  their  approaches  to  HSI 
evaluation.  Their  emphasis  is  empirical  and  analytical. 
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HSI  Domains  Addressed: 

These  methods  can  potentially  address  all  of  the  HSI  domains.  They  have  primarily  been  developed  with 
the  human  factors  domain  in  mind.  However  they  also  can  be  useful  in  surveying  the  other  domains,  if 
one  can  define  criteria  of  satisfaction  and  performance  in  the  domain  of  interest  and  measures  or 
performance  indexes  to  structure  and  guide  the  survey. 

Questions  Addressed: 

The  primary  questions  center  on  the  acceptability  of  a  system  or  subsystem  as  indexed  against  the  criteria 
and  performance  measures  developed  by  the  analysis  and  evaluation  team.  These  will  vary  with  the  HSI 
domain  under  evaluation.  As  an  example  of  one  area  of  concern  in  the  human  factors  domain,  a  question 
that  might  be  addressed  with  the  cognitive  walkthrough  analysis  could  be: 

Workflow:  For  new  systems,  are  there  any  procedures  in  which  the  human  operator  is  required  to 
engage  in  contradictory,  competing,  or  distracting  workflow  “paths?” 

Thus,  the  area  being  addressed  by  the  cognitive  walkthrough  is  that  of  workflow.  The  question  above  is 
one  instance  of  the  workflow  consideration.  There  would  be  others,  as  well  as  other  areas  to  be  addressed. 

Content  Modeled: 

These  analysis  methods  will  address  whatever  content  systems  analysts  believe  are  important  to  a 
thorough  analysis  of  the  system  from  the  HSI  point  of  view.  Although  there  is  no  modeling  of  the  content 
surveyed  in  these  analyses,  the  methods  typically  are  used  to  survey  and  analyze  areas  such  as  behavioral, 
operational  and  cognitive  competencies  required  for  proficient  system  performance;  workflow  structure, 
distractions,  interruptions  and  other  potential  disruptions  to  proficient  system  performance;  usability 
issues  surrounding  visual,  auditory  and  psycho-motor  interfaces  in  the  system;  safety  issues;  ergonomic 
and  anthropometric  issues  and  acceptability; 

Model  Granularity: 

These  tools  exist  at  a  low  level  of  granularity,  in  that  they  consist  of  survey-level  analyses  of  human 
interaction  “checkpoints”  with  the  system  under  analysis.  Their  emphasis  is  strictly  on  outcome 
measurements  of  interaction.  They  do  not  support  any  modeling  or  detailed  analysis  of  process. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Form  of  tool  output  is  survey  data  on  interaction  problems  observed  or  inferred  through  analysis  of  the 
system.  Analysis  and  output  is  communicated  primarily  through  the  language  of  interaction 
characteristics  and  errors.  Taxonomic  conventions  are  up  to  the  analyst  but  usually  consist  of  a 
combination  of  interaction  styles,  error  taxonomy,  actions  taken  by  users  of  the  system  and  a  grading 
method  used  to  rank  the  severity  of  findings.  As  an  example,  Hale  (1998)  developed  an  error  taxonomy 
for  use  with  HCI  analyses  that  consisted  of  1 1  cognitive  error  categories  observed  over  a  series  of 
interactions  with  different  types  of  software-based  systems.  These  were  communicated  to  software 
developers  through  ethnographic  observations,  interface  evaluation  surveys  and  formal  usability 
evaluations.  Error  categories  were  ranked  according  to  severity  of  impact  on  system  functionality  and 
usability. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Integration  depends  on  the  results  of  each  analysis  being  communicated  effectively  to  the  systems 
engineering  and  software  development  functions  in  the  acquisition  process. 
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Computer  Language  and  Interface  Support: 

These  are  paper  and  pencil  tools  with  no  explicit  software  or  interface  support,  other  than  that  available 
through  standard  office  applications. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

No  explicit  assumptions  regarding  risk.  Any  assumptions  used  must  be  developed  by  the  analyst. 

Error  Definitions/Taxonomy: 

N/A 

Risk  Computation/Mitigation  Methodology: 

N/A 

Traceability 

While  there  are  no  explicit  relationships  to  acquisition  requirements  inherent  in  these  tools,  analysts  can 
develop  a  traceability  matrix  that  correlates  findings  to  system  requirements.  This  correlation  would  serve 
as  a  basis  for  the  development  of  severity  ratings  to  accompany  the  usability  and  error  findings. 

These  tools  are  used  in  support  of  the  performance  assurance  phase  of  development.  In  addition,  if  a 
traceability  matrix  is  developed  during  analysis,  this  matrix  can  be  used  in  support  of  requirements  review. 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Reliability,  Validity,  and  Platform  Requirements: 

Reliability  estimates  will  be  the  responsibility  of  the  analyst,  as  most  tools  in  this  category  are  somewhat 
custom  developed.  In  general,  the  validity  of  these  methods  has  been  shown  to  be  high  over  a  large  range 
of  applications.  The  only  platform  requirement  for  the  tools  is  that  of  general  office  applications  to 
support  development  of  the  tools  themselves  and  data  compilation/analysis. 

Analysis  Utilities  and  Interface  Support: 

No  analysis  utilities  specific  to  any  of  the  tool  categories.  Descriptive  statistics  typically  will  be  the  only 
analyses  carried  out.  Analyst-developed  severity  measures  might  be  used  to  facilitate  requirements 
review. 

Availability,  Cost,  and  Contact  Information: 

These  tools  are  freely  available  at  a  number  of  sites  accessible  on  the  internet.  Representative  examples 
include: 

http://coven.lancs.ac.Uk/4/deliverables/del37e.pdf 

http://swiki.cs. Colorado. edu:3232/dlc-2002/uploads/6/dist-cogn-feb20.pdf 

https://dspace.ucalgary.ca/bitstream/1880/46646/l/2008-904-17.pdf 

http://human-factors.arc.nasa.gov/ihi/research  groups/air-ground-integration/publication  papers/Sm!999- 

CockpitWalk.pdf 
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E.6  NIOSH  Lift  Equation 

Description 

This  calculation  is  used  for  lifting  or  lowering  tasks  done  by  one  person  without  mechanical  assistance.  The 
product  of  the  NIOSH  lift  equation  is  the  Recommended  Weight  Limit  (RWL).  The  RWL  considers  the 
horizontal  parameters  of  the  lift,  the  vertical  height  at  the  start  of  the  lift,  the  vertical  distance  traveled 
during  the  lift,  the  angle  traveled,  lift  frequency,  and  a  qualifier  for  rating  the  comfort  of  the  hand  holds. 

The  maximum  RWL  with  all  factors  being  optimal  is  5 1  pounds. 

Tool/Method  Content 

Theoretical  Assumptions: 

NIOSH  assumes  that  lifting  and  lowering  tasks  carry  the  same  risk  of  lower  back  injury.  There  are  a 
number  of  limitations  when  using  the  NIOSH  lift  equation.  The  lift  must  be  a  two-handed  lift  and  the 
work  shift  must  be  less  than  eight  hours.  The  lift  equation  cannot  be  applied  to  a  task  in  an  environment 
that  anticipates  slips  or  falls,  or  environments  with  unfavorable  temperature  and  humidity  conditions.  The 
lift  task  must  be  completed  while  standing.  The  equation  cannot  be  applied  if  the  lift  occurs  while  the 
worker  is  seated  or  kneeling  or  in  a  workspace  that  restricts  movement.  The  equation 
load  is  carried  more  than  2  steps,  or  is  pushed/pulled/shoveled  as  in  a  wheelbarrow  or 
these  conditions  apply,  the  assumptions  made  my  NIOSH  in  formulating  the  equation 
are  other  tools,  Snook  &  Cirello  tables  for  example,  that  can  be  used  instead. 

HSI  Domains  Addressed: 

This  tool  addresses  the  safety  domain. 

Questions  Addressed: 

•  Is  the  load  too  heavy  for  the  parameters  of  this  lift? 

•  If  so,  which  parameter  is  causing  the  recommended  weight  limit  to  be  low? 

•  How  often  can  this  lift  be  performed  per  minute/hour? 

Content  Modeled: 

•  Load  weight,  shape,  handles 

•  Vertical  height  and  horizontal  distance  of  start  and  end  points  of  the  lift 

•  Frequency  of  lift 

•  Twisting  while  lifting 

Model  Granularity: 

This  tool  is  easy  to  use  and  can  be  done  using  worksheets  and 
There  are  also  COTS  software  packages  that  allow  the  user  to 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  RWL  is  used  to  determine  the  Lift  Index  (LI)  of  the  task, 
given  the  actual  load  of  the  lift  versus  the  RWL. 


tables,  or  in  an  electronic  spreadsheet, 
use  this  with  a  GUI. 


The  LI  is  an  estimate  of  physical  stress 


does  not  apply  if  the 
dolly.  If  any  of 
do  not  apply.  There 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

There  are  a  number  of  limitations  regarding  the  application  of  this  lift.  To  apply  this  equation  for  a  two- 
person  lift,  or  a  carry,  for  example,  would  invalidate  the  assumptions  of  the  equation. 

Traceability 

Output  categories  of  the  HFW  tool  set  include  errors  and  their  probabilities  of  occurrence,  factors  leading  to 
the  appearance  of  errors  and  estimates  of  the  prominence  of  these  factors  in  error  production,  mitigation 
strategies,  and  likely  error  consequences.  This  information  can  be  used  to  assist  in  analysis  of  technology 
alternatives,  trade-off  analyses,  and  analyses  of  system  effectiveness. 

Development  Steps  Supported: 

•  Function  analysis 

•  Task  design 

•  Performance/workload/training  estimation 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Availability,  Cost,  and  Contact  Information: 

“Revised  NIOSH  Equation  for  the  Design  and  Evaluation  of  Manual  Lifting  Tasks”  Water,  Putz- 
Anderson,  Garg,  and  Fine,  1993. 

E.7  Rapid  Upper  Limb  Assessment  (RULA) 

Description 

RULA  is  a  survey  method  developed  for  use  in  ergonomic  investigation  of  workplaces  where  work  related 
upper  limb  disorders  are  reported.  RULA  is  a  screening  tool  that  assesses  biomechanical  and  postural  load 
on  the  whole  body  with  particular  attention  to  the  neck,  trunk  and  upper  limbs. 

Tool/Method  Content 

Theoretical  Assumptions: 

Depending  on  the  type  of  study,  an  analysis  may  be  done  on  the  longest  held  posture  or  what  appears  to  be 
the  work  posture,  or  postures,  adopted. 

HSI  Domains  Addressed: 

This  tool  addresses  the  Workstation  Design  and  Physical  Ergonomics  domains. 

Questions  Addressed: 

•  Is  the  risk  of  upper  body  cumulative  trauma  disorders  acceptable? 

Content  Modeled: 

•  Body  postures  (both  upper  limb  and  whole  body) 

•  Static  or  repetitive  motion 

•  Load  size/force 
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An  example  of  a  RULA  Employee  Assessment  Worksheet  is  shown  below.  Limb,  trunk  and  muscle  scores 
are  mapped  to  a  set  of  tables,  which  are  combined  to  arrive  at  an  acceptability  score  for  the  workplace  in 
question. 

Model  Granularity: 

The  model  relies  heavily  on  the  quality  of  the  task  breakdown  and  input  in  to  the  analysis. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

RULA  output  is  a  single  digit  Final  Score.  However,  in  calculating  the  Final  Score,  individual  body  parts 
are  given  a  score,  as  are  muscle  use  and  load  size.  These  scores  can  be  compared  to  provide  a  picture  of 
relative  risk. 


RULA  Employee  Assessment  Worksheet  aansl  Li.n  jfV£A;  j  garvcp  metherf  far  i»e  investiffxttQn  of  Uretti -refried  upper  Pali  dHerdan.  tHAtinxie*  £  Cariatt  AppPed  Sr^eaoruki  19P3, 24t2},  91-99 


A,  Arm  and  Wrist  Analysis 

Step  1:  Locate  Upper  Arm  Position 
-1  +2 


20' 


2tr 


ilep  1  a:  Adjust. . . 

If  sbouldar  is  raised:  -1 

If  jpper  arm  is  abducted  +1 

If  arm  is  supported  nr  persoc  is  Lesamng:  - 1 

Step  2:  Locale  Lower  Arm  Position: 

+1  f~)  ^  - 


Upper  Arm 
Gears 


Lower  Arm 

Soars- 

Step  2a:  Adjust 

If  either  arm  is  work  mg  acr&ss  nidlme  or  out  to  side  of  body:  Add  +1 


SfepJ:  Locate  Wrist  Position: 


+3 


Viepia:  Adjust... 

If  wnst  is  tent  &ox  midlbe:  Add  +1 

Step-1:  Wiul Twist 
If  wrist  is  twisted  m  mid-range:  +1 
If  wrist  is  a:  or  neat  end  of  ranee  +2 


iftfll 

i  j 

JAii 

Wrist  Swre- 

Whist  TwsL 
Score 


Step  5:  Look-up  Posture  Score  in  Table  A: 
Usicg  value  =  from  steps  1 4  above.  Isxaie  score  it 
Table  A 

Step  fi:  Add  Muscle  Use  Score 
Ifpost.ite  mamly  stalk  (i.e.  Led d.:- 1 C1  mi nv i re s; . 

Or  if  acton  repe&ted  occurs  4X  per  ntimite:  +1 

Step  7:  Add  Force/Load  Score 
If  load  .4.-  lbs  (iniamitiec'):  -S 
If  load  4.4  to  12  lbs  ( intend  tieci):  -1 
If  load -.4  to  22  lbs  (static  or  repeated):  +2 
If  mote  than  22  lbs  or  repeated  or  shocks:  -3 

Step  S:  Find  Row  in  Table  C 
Add  values  from  steps  5-7  to  nbfam 
Wri  sr  and  Ann  Score.  Fbd  row  b  Table  C. 


SCORES 


TabeA:  Wrist  Posture  Score 

1 

2 

3 

4 

Upper 

Ann 

_cwer 

Amt 

Vitisl 

T^Sl 

"Afist 

T*is1 

Vital 

?*ist 

Wist 
Tv  si 
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2 

1 

2 

1 

2 

1 

2 

1 

1 

2 

2 

2 

2 

3 

3 

3 

1 

2 

2 

2 

2 

2 

3 

3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

4 

4 

1 

2 

3 

3 

3 

3 

4 

4 

4 

2 

2 

3 

3 

3. 

3 

3 

4 

4 

4 

3 

3 

4 

4 

4 

4 

5 

5 

1 

3 

3 

4 

4 

4 

5 

5 

3 

2 

3 

4 

4 

4 

4 

5 

5 

3 

4 

4 

4 

4 

5 

5 

5 

1 

4 

4 

4 

4 

4 

5 

5 

5 

4 

2 

4 

4 

5 

5 

5 

3 

4 

4 

0 

5 

5 

0 

S 

1 

c 

a 

5 

0 

5 

e 

6 

7 

5 

2 

5 

0 

0 

0 

6 

7 

7 

7 

a 

6 

0 

0 

7 

7 

7 

7 

E 

i 

7 

7 

7 

7 

7 

fl 

B 

9 

6 

2 

3 

8 

8 

S 

B 

9 

9 

9 

3 

9 

9 

9 

9 

9 

9 

9 

9 

Table  C:  Neck  burn  o"d  leg  scoie 


1 

1 

2 

3 

3 

4 

B 

B 

2 

~~2~ 

: 

~T 

~4~ 

~r 

5 

5 

3 

■3 

3 

3 

4 

4 

5 

6 

4 

3 

3 

3 

4 

E 

3 

6 

5 

4 

A 

± 

3 

~r 

7 

7 

0 

4 

A 

~T 

6 
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7 

7 

7 

5 

S 

a 

6 

■ 

7 

7 

8+ 

5 

i 

n 

Scoring:  (final  scare  fiom  Table  C) 

1  or  2  =  acceptable  posture 

3  or  4  =  fuithei  investigation.  c"dvge  may  ne  neocod 
5  orfc  =  fuither  investigation.  c°.ange  smt 
7  =  investigate  and  imntement  c%ange 


Wist  i  Am  Scare 


Final  Scare 


B.  Neckr  Trunk  and  Leg  Analysis 


Step  0:  Locate  Neck  Position: 

o  ur  , ,,  id-21]- 


Scot  So:  Adjust. . . 

If  neck  is  twisted:  -1 
I:  neck  is  side  heeding:  +1 


Step  ID:  Locate  Trunk  Position: 


Step  ISa.  Adjust... 

If  trunk  is  twisted:  -1 
If  trunk  is  side  tending:  -1 


If  lacs  acd  feet  ate  supported:  -  L 

If  dm:  +2 

Hick 

tadju 

Task  3:  Trurt  =sstune  Eccxte 

Leg  Score 

mm 

D 

^J5 

Lh;s 

Le 

j: 

Le 

is 

Le 

j: 

Basra 

1 

2 

2 

1 

2 

1 

: 

2 

□ 

X 

3 

3 

A 

j 

a 

E 

e 

t 

7 

i 

i 

□ 

X 

3 

A 

a 

t 

3 

E 

T 

T 

7 

a. 

3 

□ 

y 

4 

A 

a 

E 

E 

T 

? 

7 

A 

a 

a 

C 

4 

T 

' 

T 

T 

T 

4 

ft 

i 

T 

T 

r 

T 

T 

a 

i 

A 

E 

A 

a 

1 

# 

9 

9 

ft 

Step  12:  Look-up  Posture  Score  in  T able  B: 

Us  mg  values  from  stepri  9-11  shove, 
locate  score  b  Table  B 

Step  13:  Add  Muscle  Use  Score 

If  ps  stur-e  mainly  slatic  (:.e  held  :  1 Z  nunutes). 

Or  if  actioc  Treated  occurs  4X  pet  nurutte  +1 

Step  14:  Add  Force.'Lond  Score 
If  laad  =:  -.4  Lbs  fintenutteut):  +Q 
If  load  4  4  to  22  Lbs  (iutenmtcait):  +1 
If  load  4.4 10  22  Lbs  (state  Dr  repeated':  -1 
If  more  thac  22  Lbs  or  repeated  or  shocks:  -3 

Step  IS:  Find  Column  iu  Table  C 

Add  values  from  steps  12-14  to  obtain 

Neck.  Trunk  ?nd  Lee  Score.  Ficd  Column ic Table  C. 


Task  name: 


Reviewer: 


Date: 


TJxs  fira/  /s,DTDVxisd  without  warranty .  The  author  has  p?-  .kra  Ctct  tool  as  a  swrsat  ’ream  for  aopiyia?  ths  concepts  orou idhtfia  . 


iff  tteese  Co.wa/iPv..  Stv 


promdbLJ'iy  PraoLta;  Lroom.-ni^ 
rhar&B-gjsppsraart. ccvtt  fSJS)  44-t-2667 


Figure  E-3.  Rapid  Upper  Limb  Assessment  worksheet. 
(http://personal.health.usf.edu/tbemard/HollowHills/RULA.pdf) 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

This  tool  is  to  be  used  in  conjunction  with  other  analysis  methods  and  should  not  be  use  alone  to 
determine  risk. 

Traceability 

The  analysis  can  be  conducted  before  and  after  an  intervention  to  demonstrate  that  the  intervention  has 
worked  to  lower  the  risk  of  injury. 

Development  Steps  Supported: 

•  Concept  definition 

•  Requirements  analysis 

•  Task  design 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Availability,  Cost,  and  Contact  Information: 

McAtamney,  L  &  Corlett.  “RULA:  a  survey  method  for  the  investigation  of  work-related  upper  limb 
disorders.”  Applied  Ergonomics. { 24)91-99.  1993. 

Online  RULA  tool  incorporated  into:  Ergolntelligence  by  NexGen,  ErgoSure  Pro  by  Magnitude  Inc. 
Also  versions  by  COPE  and  Humanics  are  commercially  available. 


E.8  Ship  System  Human  Systems  Integration  for  Affordability  and  Performance 
Engineering  (Ship-SHAPE)  Tool  Set 

Description 

Ship-SHAPE  is  an  adaptation  of  the  Integrated  Decision/Engineering  Aid  (IDEA)  tool  set  developed  by 
Carlow  for  the  Army’s  Human  Research  and  Engineering  Directorate,  Naval  Sea  Systems  Command,  the 
Navy’s  Space  and  Warfare  Systems  Command  (SPAWAR),  and  DARPA.  Ship-SHAPE  is  a  set  of 
automated  processes,  tools,  and  data  bases  developed  specifically  to  enable  HSI  analysts  in  the  Navy  and  in 
the  commercial  ship  building  and  maritime  system  arena  to  meet  HSI  requirements  as  contained  in  the  DoD 
5000  series,  the  Defense  Acquisition  Deskbook,  Naval  Sea  Systems  Command  Instruction  3900.8,  MIL- 
STD-1472,  MIL-HDBK-46855,  ASTM-1 166  and  ASTM-1337. 


Ship-SHAPE  Automated  Human  Systems  Integration  (HSI)  tools  include: 

a)  An  HSI  Process  Tool; 

b)  A  mission/function  analysis  tool  and  scenario  generator  (IMAGE); 

c)  Comparability  Analysis  (I-CAN)  tool  which  supports  the  identification  of  high  driver 
tasks/conditions  and  lessons  learned  from  predecessor  systems; 

d)  A  function  allocation  tool  to  support  investigation  of  alternate  feasible  roles  of  the  human:  Role  of 
Man  and  Automation  (ROMAN) 

e)  The  HSI  Assessment  tool  (ASSESS)  for  assessing  technology,  affordability  and  risk  associated  with 
design  concepts; 

f)  A  Task  Analysis  Tool  (I-TASK)  based  on  MIL-H-46855  and  MIL-STD-1478; 

g)  A  Simulation  for  Workload  Assessment  and  Modeling  (SIMWAM)  tool  for  assessing  multi-operator 
task  network  impacts  on  human  performance  and  workload; 
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h)  A  tool  which  supports  planning  an  HSI  effort  by  tracking  project  tasks,  personnel  hours,  task  status, 
deliverables  with  due  dates,  called  HSI  Planning  (I-PLAN); 

i)  A  usability  testing  tool  (CUTTER)  to  support  all  phases  of  usability  testing.  This  tool  consists  of 
three  modules:  (1)  a  test  preparation  and  planning  support  module  (2)  a  data  logging  and  data 
analysis  module  and  (3)  an  interface  evaluation  guideline  module; 

k)  An  Integrated  NDI  Selection/ Assessment  Tool  (INDI) 


Tool/Method  Content 

Theoretical  Assumptions: 

The  guiding  principle  behind  the  design  of  the  Ship-SHAPE  software  is  that  the  HSI  analyst  should  have 
at  his  or  her  fingertips  all  of  the  guidance,  instructions,  processes,  procedures,  methods,  tools,  and  data 
needed  to  conduct  a  timely  and  complete  HSI  effort.  The  elements  of  the  Ship-SHAPE  system  are:  the 
HFE  process  for  ships;  an  integrated  HFE  information  system;  automated  HFE  tools;  and  a  report 
generator  for  producing  HFE  plans  and  reports. 

HSI  Domains  Addressed: 

•  HFE 

•  Training 

•  Manpower 

•  Personnel 

•  Safety 

Questions  Addressed: 

These  tools  address  planning  and  administrative  issues  involved  in  HSI,  in  addition  to  many  of  the 
technical  issues  in  the  HFE,  personnel,  manpower  and  safety  domains.  Cost  issues  also  are  addressed. 

Content  Modeled: 

The  tools  allow  modeling  and  analysis  of  task  structure  and  organization,  workload,  allocation  questions, 
design  alternatives,  teaming  and  team  structure  issues,  performance  levels,  and  error/risk  concerns. 

Model  Granularity: 

The  granularity  of  these  models  seems  to  be  moderate.  Models  developed  under  the  support  of  most  of 
the  tools  in  this  set  are  descriptive  rather  than  quantitative  or  executable.  Most  descriptions  will  be  at  an 
operational,  rather  than  at  a  process,  level. 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  is  available  in  a  wide  variety  of  forms,  depending  on  the  tool  in  use.  The  outputs  include: 

•  Display  of  results  of  a  mission  function  analysis  in  tabular  or  flowchart  form; 

•  Guidelines  on  performance  of  specific  HSI  activities  in  the  context  of  ship  operations; 

•  Tasks  that  have  a  likelihood  of  significantly  affecting  operational  effectiveness; 

•  Function  allocations  and  roles  of  humans  in  the  overall  system; 

•  Comparative  assessments  of  HSI  design  concepts  and  alternatives; 

•  Task  analysis  data  for  subsequent  design  and  engineering  analyses; 

•  Mission  times,  task  completions,  task  start  and  end  times,  time  spent  per  task  per  operator,  and 
operator  utilization; 

•  Project  planning  documents,  status  reports,  hours  and  tasks  by  persons  and  by  project  month. 
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Terminology  and  taxonomic  conventions  are  those  of  traditional  human  factors  and  HSI  engineering. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

It  will  be  the  responsibility  of  the  HSI  analyst  to  communicate  these  to  the  larger  system  development 
process. 

Computer  Language  and  Interface  Support: 

MS  Office  is  required  for  most  of  these  tools.  Other  language  requirements  include  HyperCard,  Filemaker 
Pro,  and  an  internet  browser. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Not  known. 

Error  Definitions/Taxonomy: 

No  known. 

Risk  Computation/Mitigation  Methodology: 

Not  known. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

The  wide  variety  of  outputs  provided  by  these  tools  can  be  used  to  support  several  acquisition  requirement 
categories:  Analysis  of  technology  alternatives,  evaluations  of  system  effectiveness,  functional  allocation, 
and  CONOPS  development.  The  administrative  tools  can  support  decision  criteria  definition  and 
evaluation. 

Development  Steps  Supported: 

•  Concept  definition 

•  Requirements  analysis 

•  Function  analysis 

•  Function  allocation 

•  Task  design 

•  Performance/workload/training  estimation 

•  Personnel  selection 

•  Training  development 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

All  of  the  tools  have  been  formally  validated  and  have  been  used  in  a  number  of  programs  conducted  by 
the  US  Navy.  The  tools  will  run  on  either  Windows  machines  or  Apple  Macintosh  platforms. 
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Analysis  Utilities  and  Interface  Support: 

All  of  the  tools  contain  analysis  utilities  within  the  tools  themselves.  Interface  support  is  provided 
primarily  by  the  Office  software  upon  which  each  tool  relies. 

Availability,  Cost,  and  Contact  Information: 

The  Ship-SHAPE  web  site  states  that  all  tools  are  readily  available.  However,  the  platform  requirement 
descriptions  imply  that  these  tools  were  developed  some  time  ago. 

Carlow  International  Incorporated 

Thomas  B.  Malone,  President 

tbmalone@carlow.com 

20856  Waterbeach  Place 

PO  Box  650457 

Potomac  Falls,  VA  20165  USA 

703.444.4666 

http://carlow.com/index.html 

E.9  Risk  Management  Toolkit 

Description 

The  risk  management  toolkit  is  a  management  and  CMMI  process  tool  set  designed  to  allow  development 
teams  to  define,  track  and  mitigate  project  risks.  In  addition  to  the  processes  and  procedures  that  have  been 
defined  as  part  of  this  tool  kit,  there  are  three  software  packages  designed  to  assist  program  managers  and 
development  teams  with  system  development. 

RiskNav®  is  a  well-tested  tool  developed  by  MITRE  to  facilitate  the  risk  process  and  help  program 
managers  manage  their  risk  space.  RiskNav  lets  you  collect,  analyze,  prioritize,  monitor,  and  visualize  risk 
information  in  a  collaborative  fashion.  This  tool  provides  three  dimensions  of  information  graphically  (risk 
priority,  probability,  and  the  mitigation/management  status). 

RiskNav  is  a  well-tested  tool  developed  by  MITRE  to  facilitate  the  risk  process  and  help  program  managers 
manage  their  risk  space.  RiskNav  lets  you  collect,  analyze,  prioritize,  monitor,  and  visualize  risk 
information  in  a  collaborative  fashion.  This  tool  provides  three  dimensions  of  information  graphically  (risk 
priority,  probability,  and  the  mitigation/management  status). 

Risk  Radar  is  a  risk  management  database  to  help  project  managers  identify,  prioritize,  and  communicate 
project  risks  in  a  flexible  and  easy-to-use  form.  Risk  Radar  provides  standard  database  functions  to  add  and 
delete  risks,  as  well  as  specialized  functions  for  prioritizing  and  retiring  project  risks.  Each  risk  can  have  a 
user-defined  risk  management  plan  and  a  log  of  historical  events.  A  set  of  standard  short-  and  long-form 
reports  can  be  easily  generated  to  share  project  risk  information  with  all  members  of  the  development  team. 
The  number  of  risks  in  each  probability/impact  category  by  time  frame  can  be  displayed,  which  allows  the 
user  to  drill  down  through  the  data  to  uncover  increasing  levels  of  detail.  Risk  Radar  allows  the  user  the 
flexibility  of  using  automatic  sorting  in  addition  to  manually  moving  risks  up  and  down  in  setting  priority 
rank. 
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Tool/Method  Content 

Theoretical  Assumptions: 

This  is  a  management  toolkit  derived  from  the  Camegie-Mellon  University  Software  Engineering  Institute 
(SEI).  As  such,  there  are  minimal  theoretical  assumptions  in  the  development  and  use  of  the  toolkit. 

What  theoretical  assumptions  are  present  are  based  on  the  notion  that  human  errors  occupy  the  status  of 
causal  agents  in  analyses  of  system  failures. 

HSI  Domains  Addressed: 

All  domains  are  addressed  by  this  toolkit.  Some  domains  are  addressed  indirectly  through  a  propagation 
of  analysis  results  into  systems  engineering  analysis  and  design  activities. 

Questions  Addressed: 

The  toolkit  addresses  questions  of  acquisition  management  in  the  following  areas: 

•  Acquisition 

•  Contracting 

•  Cost 

•  Environmental  aspects 

•  Funding 

•  Health  hazards 

•  Human  factors 

•  Logistics  planning 

•  Manpower 

•  Personnel 

•  Requirements 

•  Resources 

•  Safety 

•  Scheduling 

•  Software  development 

•  Survivability 

•  Systems  engineering 

•  Training 

Content  Modeled: 

The  primary  content  modeled  by  the  toolkit,  in  each  of  the  areas  outlined  above,  includes  determination  of 
risk  factors,  the  relationships  of  these  factors  to  program  outcomes  and  mitigation  strategies  that  can  be 
applied  to  the  risks. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Output  comes  primarily  in  the  form  of  risk  factors  and  mitigation  strategies. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

This  toolkit  is  intended  to  be  a  systems  engineering  method.  The  areas  of  interest  in  the  HSI  domain  are 
included  in  the  toolkit. 
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Computer  Language  and  Interface  Support: 

Not  applicable  except  in  the  case  of  the  three  tools  outlined  in  the  description  section  above.  RiskNav® 
uses  a  weighted  averaging  model  and  tabular  format  to  analyze  program  risks.  Risk  Matrix  also  uses  a 
tabular  interface  based  on  Excel  spreadsheets.  Risk  Radar  uses  a  forms-based  approach  to  capture 
information  that  is  entered  into  a  database  for  tracking  and  resolution. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risks,  in  each  of  the  areas  outlined  above,  can  be  identified  through  a  subjective,  analytical  process, 
tabulated  and  resolved  through  weighted  averaging  and  other  methods. 

Error  Definitions/Taxonomy: 

This  toolkit  contains  no  standard  error  taxonomy. 

Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

All  output  categories  concern  risks,  probabilities  of  occurrence,  severity  associated  with  risk  occurring, 
and  mitigation  strategies.  Since  this  toolkit  is  a  program  management  tool,  the  acquisition  requirements 
addressed  by  the  toolkit  are  enumerated  in  the  above  section  on  “questions  addressed.” 

Development  Steps  Supported: 

•  Requirements  analysis 

•  Requirements  review 

How  Tool  Maps  System  Requirements  to  HSI  Requirements: 

This  toolkit  should  allow  explicit  mapping  of  system  requirements  to  HSI  requirements  as  an  outgrowth  of 
the  scope  and  approach  taken  to  risk  analysis  at  the  program  level.  However,  the  toolkit  contains  no 
explicit  method  of  doing  this  other  than  providing  the  process  and  information  to  support  this  mapping. 
Actually  making  the  mapping  is  the  responsibility  of  the  development  team. 


Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

Reliability  and  validity  information  is  unavailable.  Platform  requirements  apparently  include  standard 
Windows  and  OS-X  platforms  running  Office  applications. 


Analysis  Utilities  and  Interface  Support: 

Analysis  and  interface  support  are  provided  in  the  three  software  packages  mentioned  above.  Other 
support  requirements  are  provided  through  tools  contained  on  the  platforms  used  to  run  the  tools,  e.g., 
Office  applications. 


Availability,  Cost,  and  Contact  Information: 

Availability  seems  to  come  primarily  through  the  Risk  Management  Toolkit  website.  The  Risk  Radar  tool 
is  a  third-party  development.  Availability/cost  information  for  this  tool  is  not  available.  The  toolkit  can 
be  accessed  at:  http://www.mitre.org/work/sepo/toolkits/risk/index.html 
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E.10  Jack™ 


Description 

Jack  is  an  ergonomics  tool  set  allowing  constructive  simulation  for  workstation  design  and  early  trade-off 
studies.  Areas  addressed  by  this  tool  include  reach,  vision,  injury  risk,  fatigue,  comfort  and  strength 
assessments. 


Figure  E-4.  Jack  workstation  design  tool. 


Tool/Method  Content 

Theoretical  Assumptions: 

There  are  no  theoretical  assumptions.  Jack  is  a  physical  modeling  environment. 

HSI  Domains  Addressed: 

•  HFE 

•  Habitability 

•  Personnel 


This  tool  set  will  allow  HSI  designers  to  address  issues  in  ergonomics,  workstation  design  and  layout,  and 
personnel  (through  its  ability  to  inform  designers  regarding  anthropometric  and  strength  requirements). 

Questions  Addressed: 

The  tool  set  is  focused  almost  exclusively  in  the  areas  of  anthropometry,  strength,  and  physical  layout. 
Vision  analyses  also  can  be  completed. 

Content  Modeled: 

Posture  predictions,  3D  vision  obscuration,  reflection  areas,  visibility  zones,  strength  limits,  posture, 
injury  risk,  fatigue  and  task  timing. 

Model  Granularity: 

The  tool  set  has  a  high  degree  of  granularity  in  both  modeling  and  analysis  output. 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  output  can  include  visual  viewpoints  from  the  model’s  point  of  view,  “view  cones”  that  illustrate  a 
third-person  perspective  of  what  the  model  sees,  distances  between  the  model’s  eyes  and  any  visual  object 
in  the  modeled  scene  or  system,  eye-tracking  trajectories,  reach  envelopes,  hand-to-object  distances, 
interactive  distance  measures,  model-object  collisions.  Terminology  and  conventions  are  those  of 
physical  ergonomics.  Examples  of  specification  screens  and  a  modeled  environment  are  shown  below. 
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Figure  E-5.  Jack  input  and  analysis  screens. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

Output  results  can  be  used  as  input  to  CAD  systems,  thereby  integrating  Jack  output  with  other  systems 
engineering  tools. 


Computer  Language  and  Interface  Support: 
The  system  uses  an  internal  language. 


Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

Risk  is  not  addressed  explicitly  in  this  system.  Any  assumptions  regarding  risk  will  be  implicit  in  the 
specification  of  anthropometric  and  other  physical  parameters  of  a  given  task  or  environment  being 
modeled. 


Error  Definitions/Taxonomy: 

None 

Risk  Computation/Mitigation  Methodology: 
None 
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Traceability 

Output  Categories  and  Relationships  to  Acquisition  Requirements: 

These  include  reach,  anthropometry,  strength  and  vision  data.  These  data  can  be  used  in  requirements 
analyses  and  other  design  analyses  that  are  included  in  the  development  of  PORDs  and  ORDs. 

Development  Steps  Supported: 

•  Task  design 

•  Personnel  selection 

•  Performance  assurance 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

The  tool  has  been  validated  over  a  wide  range  of  applications.  Platform  requirements  include: 

Windows  2000  or  XP 
Minimum  300  MHz  processor 
128  MB  RAM 
175  MB  free  disk  space 

Analysis  Utilities  and  Interface  Support: 

Complete  analysis  utilities  and  user  interface  are  included  with  the  tool. 

Availability,  Cost,  and  Contact  Information: 

Siemens  PLM  Software 
(800)  498-5351 
www.  siemens .  com/plm 

E.ll  Human  Factors  Workbench  (HFW) 

Description 

An  integrated  software  package  composed  of  five  analytical  tools  that  can  be  used  independently  or 
together.  Among  the  tools  in  the  set  are  the  Predictive  Human  Error  Analysis  (PHEA)  tool  and  the 
Measurement  and  Investigation  Technique  to  Reduce  Errors  (MITRE)  tool.  The  PHEA  is  used  to  predict 
potential  human  errors  and  their  consequences.  The  MITRE  tool  allows  analysts  to  assess  factors 
influencing  the  likelihood  of  errors  identified  in  the  PHEA  and  to  develop  specific  prevention  strategies  for 
mitigating  the  consequences  of  errors. 

Tool/Method  Content 

Theoretical  Assumptions: 

An  important  theoretical  assumption  of  these  tools  is  that  one  can  develop  an  exhaustive  list  of  error 
modes  and  that  these  can  be  used  to  (1)  identify  (potential)  errors  across  a  wide  range  of  operational 
situations,  (2)  estimate  error  probabilities,  (3)  relate  the  errors  to  causal  factors.  The  critical  assumption  is 
that  this  process  can  be  exhaustive,  that  is,  that  all  error  types  and  causes  can  be  identified  and  related  in  a 
quantitative  way. 
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HSI  Domains  Addressed: 

•  HFE 

•  Training 

•  Safety 

Questions  Addressed: 

•  What  errors  might  be  committed  in  the  operational  scenarios  of  interest 

•  What  causal  factors  can  be  identified  to  “explain”  the  errors  inherent  in  an  operational  scenario  of 
interest 

•  What  are  the  likelihoods  of  errors  in  each  of  the  categories  of  interest 

•  What  is  the  significance  of  each  “causal  factor”  on  each  error  category 

Content  Modeled: 

•  Tasks  and  task  sequences 

•  Error  types 

•  Causal  factors  influencing  the  probabilities  of  errors  in  each  category 

•  Error  probabilities 

•  Error  commission  consequences 

•  Error  mitigation  strategies 

A  task  analysis  screenshot  and  associated  error  taxonomy  are  shown  in  Figure  E-6  and  Figure  E-7  (below). 
The  system  provides  support  for  development  of  both  of  these  design  artifacts. 
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Figure  E-6.  Section  of  a  Human  Factors  Workbench  task  analysis  for  taking  a  propane  tank  out  of  service. 

(from  www.humanreliability.com) 
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Action  Errors 

Al 

Operation  too  long  /  short 

A2 

Operation  mistimed 

A3 

Operation  in  wrong  direction 

A4 

Operation  too  little  /  too  much 

A5 

Operation  too  fast  /  too  slow 

A6 

Misalign 

A7 

Right  operation  on  wrong  object 

A8 

Wrong  operation  on  right  object 

A9 

Operation  omitted 

A10 

Operation  incomplete 

All 

Operation  too  early  /  late 

A12 

Operation  in  wrong  order 

A13 

Misplacement 

Checking  Errors 

Cl 

Check  omitted 

C2 

Check  incomplete 

C3 

Right  check  on  wrong  object 

C4 

Wrong  check  on  right  object 

C5 

Check  too  early  /  late 

Information  Retrieval  Errors 

R1 

Information  not  obtained 

R2 

Wrong  information  obtained 

R3 

Information  retrieval  incomplete 

R4 

Information  incorrectly  interpreted 

Information  Communication  Errors 

11 

Information  not  communicated 

12 

Wrong  information  communicated 

13 

Information  communication  incomplete 

14 

Information  communication  unclear 

Selection  Errors 

SI 

Selection  omitted 

S2 

Wrong  selection  made 

Planning  Errors 

PI 

Plan  incorrect  because  of  misdiagnosis 

P2 

Diagnosis  correct  but  wrong  action  plan  formulated 

Figure  E-7.  Predictive  human  error  analysis  scheme  used  in  Human  Factors  Workbench. 

Model  Granularity: 

The  PHEA  tool  provides  an  a  priori  classification  of  observable  failure  modes.  The  tool  authors  claim  that 
this  list  is  exhaustive.  Currently,  the  list  consists  of  30  error  modes,  organized  into  six  error  categories. 
These  error  modes  reside  at  a  moderate  level  of  granularity,  stated  operationally  at  a  level  allowing 
enumeration  of  individual  “errors”  but  not  allowing  any  insight  into  process  or  microscopic  detail. 


Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

These  tools  provide  a  variety  of  outputs  that  can  be  used  in  system  development: 

•  PHEA  analysis  produces  a  spreadsheet  output  compiling  errors,  possible  accident  hazard 

consequences,  risk  control  measures  and  performance  influencing  factors 

•  Based  on  an  influence  diagram  model,  MITRE  outputs  to  the  design  team  evaluation  results  of  the 

factors  affecting  human  errors  in  scenarios  of  interest.  These  include  histograms  relating  error 
factors  to  overall  failure  assessment  likelihoods 

•  Risk  reduction  strategies  based  on  relative  cost  and  effectiveness 
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•  Overall  “quality”  of  factors  affecting  error  probabilities.  The  MITRE  tool  calculates  an  index 

assessing  this  “quality”  and  provides  estimates  of  error  probabilities  under  selected  operating 
conditions.  It  is  not  clear  from  the  literature  available  what  is  meant  by  the  term  “quality”  in 
connection  with  these  computations 

The  tools  used  in  the  HFW  rely  on  the  standard  terminology  of  errors  and  error  probabilities.  Taxonomic 
conventions  are  those  of  standard  accident  investigation. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 

There  are  no  explicit  SE  integration  methods  contained  in  this  tool  set.  However,  the  use  of  standard 
accident  investigation  and  error  probability  conventions  should  improve  the  ease  with  which  these  tools 
and  their  results  can  be  integrated  into  larger  SE  trade-off  processes. 

Computer  Language  and  Interface  Support: 

The  tool  set  contains  a  full  range  of  interfaces  for  data  entry,  analysis  and  reporting. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

These  tools  are  based  on  the  following  assumptions: 

•  Risk  is  an  outcome  of  system  failures  resulting  from  errors  committed  by  human  operators, 

•  Errors  are  committed  when  factors  are  present  that  create  the  conditions  under  which  the  errors  can 

arise, 

•  Factors  leading  to  error  generation  result  from  system  characteristics  and  design  commitments  that 

can  be  detected  through  analysis  and  corrected  with  proactive  design  decisions  or  retroactive 
changes  to  system  operations,  structure  or  organization, 

•  Error  probabilities  can  be  estimated,  and  mitigation  strategies  devised  through  an  analysis  process 

embodied  in  the  HFW  PHEA  and  MITRE  tools. 

Error  Definitions/Taxonomy: 

Definitions  and  taxonomies  are  contained  in  the  HFW  tool  set  and  have  been  discussed  in  the  review 
comments  above. 

Risk  Computation/Mitigation  Methodology: 

For  tasks  of  interest,  potential  errors  and  their  consequences  are  identified  and  placed  in  a  spreadsheet 
along  with  mitigation  strategies  and  performance  influencing  factors.  These  factors  are  then  assessed 
through  a  tool- supported  questionnaire  and  rated  to  provide  overall  assessments  of  failure  likelihoods  at 
each  level  of  a  task  tree.  Alternative  task  reduction  strategies  are  evaluated  using  available  data  on  cost. 

The  overall  results  of  the  assessment  of  factors  contributing  to  errors  of  interest  can  then  be  calculated.  The 
tool  calculates  an  index  which  indicates  overall  quality  of  factors  in  scenarios  of  interest.  Estimates  of  error 
probabilities  under  the  conditions  of  interest  are  calculated  and  can  be  displayed  in  a  manner  similar  to  that 
shown  below  (Figure  E-8). 
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Figure  E-8.  Error  probability  estimates  from  Human  Factors  Workbench, 
(from  www.humanreliability.com) 


Traceability 

Output  categories  of  the  HFW  tool  set  include  errors  and  their  probabilities  of  occurrence,  factors  leading  to 
the  appearance  of  errors  and  estimates  of  the  prominence  of  these  factors  in  error  production,  mitigation 
strategies,  and  likely  error  consequences.  This  information  can  be  used  to  assist  in  analysis  of  technology 
alternatives,  trade-off  analyses,  and  analyses  of  system  effectiveness. 

Development  Steps  Supported: 

•  Function  analysis 

•  Task  design 

•  Performance/workload/training  estimation 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

The  HFW  tool  set  has  been  used  across  a  wide  range  of  scenarios  and  error  analysis  contexts  and  has  been 
shown  to  be  both  valid  and  reliable.  Platform  requirements  include  Windows  XP  or  Vista. 

Analysis  Utilities  and  Interface  Support: 

Each  individual  module  contains  its  own  native  analysis  utilities  and  interface  support. 
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Availability,  Cost,  and  Contact  Information: 

The  costs  for  the  HFW  modules  are  as  follows: 

•  Hierarchical  Task  Analysis  £250 

•  Predictive  Human  Error  Analysis  (PHES)  £250 

•  Measurement  and  Investigation  Technique  for  Reducing  Error  (MITRE)  £250 

Human  Reliability 
1  School  House,  Higher  Lane 
Dalton,  Lancashire  WN8  7RP 
UK 

contact@humanreliability.com 


E.12  Anthropometric  Accommodation  in  Aircraft  Cockpits:  Methodologies 

Description 

This  is  a  series  of  techniques,  focusing  solely  on  accommodation,  for  examining  aircraft  cockpits  for 
optimum  aircraft  operation.  These  techniques  address  the  variability  in  body  sizes  and  proportion  of 
potential  pilot  populations.  The  most  straightforward  use  of  the  accommodation  data  is  to  verify  design 
specifications.  If  a  cockpit  is  required  to  accommodate  a  given  range  of  body  sizes,  the  techniques  make  it 
possible  to  validate  compliance.  This  is  done  by  comparing  the  anthropometric  dimensions  in  the 
specification  to  the  results  of  the  evaluations.  Another  use  for  these  data  is  to  predict  the  fit  of  a  range  of 
body  sizes  in  a  crew  station.  Data  can  also  be  sued  to  assess  the  effects  of  expanding  the  ranges  of  body 
sizes  permitted  to  enter  pilot  training. 


Tool/Method  Content 

Theoretical  Assumptions: 

The  development  of  examination  procedures  was  based  on  aircraft  available  between  1990  and  1995 
including  the  USAF  F-16A,  C-141A,  T-37B,  T-38A,  T-1A,  and  F-22A.  Also  the  USN  T-34C,  T-44A,  T- 
45 A,  and  the  TA-4J,  the  Enhanced  Flight  Screener  (EFS)  and  the  Joint  Primary  Aircraft  Training  System 
(JPATS). 


HSI  Domains  Addressed: 

This  tool  addresses  the  workstation  and  cockpit  design  of  the  safety  domain. 

Questions  Addressed: 

•  Is  the  individual  too  large/small  to  fly  the  aircraft?  Has  the  aircraft  been  designed  to  accommodate 
particular  body  sizes? 

•  Parameters  considered:  ejection,  rudder  throw,  alternative  seated  eye  height  (the  Frankfurt  Plane), 
for  example. 


Content  Modeled: 

The  approach  relies  heavily  on  the  following  measurements:  maximum  sitting  height,  vision  from  the 
cockpit  to  the  outside  and  toward  the  instrument  panel,  static  ejection  clearances  of  the  knee,  leg,  and 
torso  with  cockpit  structures,  operational  leg  clearances  with  the  main  instrument  panel,  operational  leg 
clearance  with  the  control  stick/wheel  motion  envelope,  rudder  pedal  operation,  hand  reach  to  and 
actuation  of  controls. 
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Model  Granularity: 

The  model  takes  into  account  many  of  the  body  positions  required  to  operate  an  aircraft.  There  are  a 
number  of  human  dimensions  to  record  and  compare  against  the  norms.  However,  judgment  should  be 
applied  as  to  the  specific  aircraft  that  the  person  is  intended  to  operate. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

The  analysis  returns  usually  the  maximum  height/breadth/depth/etc  of  a  body  part  that  can  safely  operate 
the  aircraft,  maintain  a  visual  field  inside/outside  the  cockpit,  or  safely  eject..  There  are  a  number  of 
separate  analyses  for  many  aspects  of  aircraft  operation  that  need  to  be  complete  to  provide  a  more 
complete  picture  of  accommodation. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

There  are  a  number  of  limitations  regarding  the  application  of  this  lift.  To  apply  this  equation  for  a  two- 
person  lift,  or  a  carry,  for  example,  would  invalidate  the  assumptions  of  the  equation. 

Traceability 

We  have  only  limited  ability  to  predict  the  individual’s  level  of  accommodation.  This  is  true  of  all  measures 
but  especially  hand  reaches  to  controls.  When  regression  equations  are  used,  they  must  be  based  on  large 
samples.  Such  predictions  produce  “average”  values  expected  for  a  population  of  individuals  of  that  body 
size.  There  can  be  a  good  deal  of  variation  around  the  average.  If  examination  indicates  some  question 
regarding  an  individual’s  ability  to  safely  operate  the  aircraft,  a  trial  in  the  cockpit  may  be  warranted. 

Development  Steps  Supported: 

Function  analysis 
Task  design 
Requirements  review 
Personnel  selection 

Other  Evaluation  Criteria 

Learning  curve:  Shallow 

Availability,  Cost,  and  Contact  Information: 

•  Zehner,  GF  and  JA  Hudson,  Body  Size  Accommodation  in  USAF  Aircraft,  AFRL-HE-WP-TR- 
2002-0118,  United  States  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate,  Wright- 
Patterson  AFB,  OH. 

•  KW  Kennedy,  http://cockpiteval.home.att.net  Accessed  24  February  2008. 

E.13  Ergolntelligence  ™  Upper  Extremity  Assessment  (UEA) 

Description 

The  Ergolntelligence™  Upper  Extremity  Assessment  (UEA)  suite  of  tools  incorporates  a  variety  of  tools 
including  RULA,  REBA,  Strain  Index,  Occupational  Repetitive  Actions  Index  (OCRA)  and  the  Cumulative 
Trauma  Disorders  Risk  Index. 
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Tool/Method  Content 

Ergolntelligence  combines  accepted  physical  ergonomics  tools  focusing  on  the  upper  body  into  one 

integrated  software  package  with  a  graphical  user  interface. 

•  RULA  provides  a  rapid  assessment  of  the  musculoskeletal  loads  on  workers  due  to  posture, 
repetition  and  force.  It  accomplishes  these  goals  by  providing  a  “Grand  Score”  which  can  be 
compared  to  four  Action  Levels  ranging  from  “that  posture  is  acceptable”  to  “investigation  and 
changes  are  required  immediately.” 

•  REBA  (Rapid  Entire  Body  Assessment  was  specifically  designed  to  assess  various  unpredictable 
working  postures  found  in  health  care  and  other  service  industries.  REBA  provides  a  scoring  system 
for  muscle  activity  caused  by  static,  dynamic,  rapid  changing  or  unstable  postures.  The  final  REBA 
score  provides  an  action  level  with  an  indication  of  urgency. 

•  The  Strain  Index  (SI)  is  a  score  value  based  on  a  multiple  of  six  variables:  intensity  of  exertion, 
duration  of  exertion,  efforts  per  minute,  hand/wrist  posture,  speed  of  work  and  duration  of  task.  The 
output  score  determines  whether  a  job  has  a  high  risk  of  distal  upper  extremity  disorders. 

•  The  Occupational  Repetitive  Actions  Index  (OCRA)  is  a  measurement  tool  that  quantifies  the 
relationship  between  the  daily  number  of  actions  actually  performed  by  the  upper  limbs  in  repetitive 
tasks,  and  the  corresponding  number  of  recommended  actions.  OCRA  indices  >4  should  be 
considered  as  a  high-risk  job;  an  index  of  0.8  to  4  is  an  intermediate  risk  job. 

•  This  cumulative  trauma  disorder  risk  assessment  model  (CTD  Risk  Index)  for  the  upper  extremities 
represents  the  predicted  incidence  rate  for  a  cumulative  trauma  disorder.  The  model  is  unique  in  that 
it  uses  quantitative  data  such  as  hand  motion  frequencies  and  forces  together  to  obtain  a  frequency 
factor  score  that  is  reflective  of  the  strain  imposed  on  the  muscles  and  tendons  of  the  wrist.  Gross 
upper  extremity  postures  are  included  in  a  posture  factor  score  and  various  minor  job  stressors  are 
included  as  a  miscellaneous  factor  score. 

HSI  Domains  Addressed: 

This  tool  addresses  the  Workstation  Design  and  Physical  Ergonomics  domains. 

Questions  Addressed: 

•  Is  the  job/task  acceptable? 

•  How  can  this  job  be  redesigned  to  warrant  a  lower  risk  score 

Content  Modeled: 

•  Tasks  and  task  sequences 

Model  Granularity: 

The  model  relies  heavily  on  the  quality  of  the  task  breakdown  and  input  in  to  the  analysis. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Graphical  outputs  suitable  for  reports  are  standard  in  this  software  package. 

Methods  used  to  Integrate  with  SE  and  Other  Environments: 
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Computer  Language  and  Interface  Support: 

Computer  language  is  not  an  issue  with  these  proprietary  and  self-contained  tools.  The  tool  set  contains  a 
full  range  of  interfaces  for  data  entry,  analysis  and  reporting. 

Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

This  tool  is  to  be  used  in  conjunction  with  other  analysis  methods  and  should  not  be  use  alone  to 
determine  risk. 

Traceability 

Development  Steps  Supported: 

Function  analysis 
Task  design 

Performance/workload/training  estimation 

Interface  Support 

Ergolntelligence  UEA  interface  support  is  typified  by  the  screenshots  shown  below,  used  to  specify  trunk 
and  upper  limb  positioning  and  stress. 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

Ergolntelligence  runs  on  Windows  2000/NT4  and  Windows  XP 

Availability,  Cost,  and  Contact  Information: 

6600  Trans  Canada  Highway 
Suite  750 

Pointe  Claire  (Montreal),  Quebec 

Canada 

H9R  4S2 

Telephone:  (514)  685-8593 
Fax:  (514)  685-8687 
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Figure  E-9.  Ergolntelligence  upper  extremity  assessment. 

Top:  rapid  entire  body  assessment  (REBA).  Bottom:  strain  index,  (from  www.nexgenergo.com) 
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E.14  Ergolntelligence™  MMH  (Manual  Material  Handling) 

Description 

The  Ergolntelligence™  MMH  (Manual  Material  Handling)  modules  focus  on  material  handling  applications 

and  provide  an  in-depth  risk  analysis  for  low-back  injury  using  the  NIOSH  Lifting  Equation,  Biomechanics, 

Energy  Expenditure,  Mital  Tables  and  Snook  &  Ciriello  Tables. 

Tool/Method  Content 

Ergolntelligence  combines  accepted  manual  material  handling  assessment  tools  into  one  integrated  software 

package  with  a  graphical  user  interface. 

•  There  are  two  versions  for  the  NIOSH  Lifting  Equation  module  (both  incorporating  Single  and 
Multi-Task),  with  the  PRO  version  including  biomechanics  and  a  manikin  stick  figure  that  can  be 
manipulated. 

•  The  Snook  &  Ciriello  tables  can  be  used  for  the  evaluation  and  design  of  manual  handling  (lifting, 
lowering,  pushing,  pulling  and  carrying)  tasks.  Mital  tables  utilize  the  same  population  and  database 
used  in  the  Snook  tables.  However  the  values  are  adjusted  for  various  biomechanical,  physiological, 
and  epidemiological  criteria.  In  addition,  the  data  is  also  adjusted  for  factors  that  are  commonly 
found  to  be  significantly  affecting  the  maximum  acceptable  weight  of  industrial  workers. 

•  The  Job  Severity  Index  (JSI)  is  based  upon  the  ratio  of  the  required  weight  of  the  lift  to  worker 
capacity.  A  JSI  is  a  measure  of  the  musculoskeletal  strain  based  on  weight  handled,  frequency  of 
lifting,  and  a  worker’s  physical  capacity  of  lifting. 

•  The  Energy  Expenditure  module  is  based  on  the  assumption  that  a  job  can  be  divided  into  simple 
tasks  and  that  the  average  metabolic  energy  rate  of  the  job  can  be  predicted  by  knowing  the  energy 
expenditure  of  the  simple  tasks  and  the  time  duration  of  the  job.  The  Energy  Expenditure  module 
can  be  applied  to  stoop,  squat,  and  arm  lifts. 

HSI  Domains  Addressed: 

This  tool  addresses  the  safety  domain;  specifically  Workstation  Design,  Physical  Ergonomics,  Manual 

Material  Handling,  and  Task  Analysis. 

Questions  Addressed: 

•  Is  the  lift/lower/carry/push/pull  task  safe? 

•  Where  do  the  maximum  forces  occur  during  the  task? 

•  What  percentage  of  people  are  capable  of  performing  this  task? 

Content  Modeled: 

•  Tasks  and  task  sequences 

Model  Granularity: 

The  model  relies  heavily  on  the  quality  of  the  task  breakdown  and  input  in  to  the  analysis. 

Communication 

Form  of  Output,  Terminology,  and  Taxonomic  Conventions: 

Graphical  outputs  suitable  for  reports  are  standard  in  this  software  package. 
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Computer  Language  and  Interface  Support: 

Computer  language  is  not  an  issue  with  these  proprietary  and  self-contained  tools.  The  tool  set  contains  a 
full  range  of  interfaces  for  data  entry,  analysis  and  reporting,  as  shown  below. 
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Figure  E-10.  Ergolntelligence  manual  material  handling  input  and  analysis  screens. 

(from  www.nexgenergo.com") 
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Risk  and  Trade-off  Management 

Assumptions  Regarding  Risk: 

This  tool  is  to  be  used  in  conjunction  with  other  analysis  methods  and  should  not  be  use  alone  to 
determine  risk. 

Traceability 

Development  Steps  Supported: 

Function  analysis 
Task  design 

Performance/workload/training  estimation 

Other  Evaluation  Criteria 

Learning  curve:  Moderate 

Reliability,  Validity,  and  Platform  Requirements: 

Ergolntelligence  runs  on  Windows  2000/NT4  and  Windows  XP 

Availability,  Cost,  and  Contact  Information: 

6600  Trans  Canada  Highway  Suite  750 
Pointe  Claire  (Montreal),  Quebec 
Canada 
H9R  4S2 

Telephone:  (514)  685-8593 
Fax:  (514)  685-8687 
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APPENDIX  F.  A  NOTIONAL  EXAMPLE  OF  HUMAN  PERFORMANCE 
MODELING  APPLICATION 


This  appendix  provides  the  reader  with  an  example  of  how  human  performance  modeling  is  applied  in  the 
context  of  a  notional  study  that  evaluated  the  potential  mission  performance  gains  obtained  by  incorporating 
an  Unmanned  Aerial  System  (UAS)  into  the  National  Security  Cutter  (NSC)  system.  The  example  is 
provided  as  an  annotated  PowerPoint  presentation.  This  appendix  uses  the  notes  pages  from  that 
presentation. 
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3ASL 

Fjwt  SflfertCB  to  Solutions 


An  Example  of  How  Human  Performance 
Modeling  Could  Be  Applied  to  Analyze  the 
Impact  of  Unmanned  Aircraft  System  (UAS) 
Technology  on  the  National  Security  Cutter 
(NSC)  Ability  to  Detect,  Classify,  and  Identify 
Marine  Contacts 
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Human  performance  modeling  is  a  powerful  tool  available  to  HSI  professionals.  Human 
performance  models  (HPM)  provide  a  means  for  representing  human  performance 
computationally;  enabling  representation  of  the  dynamics  of  human  performance  in  complex 
systems  and  situations.  This  provides  much  better  insight  into  emergent  effects  such  as  workload 
and  situation  awareness  than  can  be  gained  from  static  task  descriptions.  When  HPM  are 
integrated  with  system  and  mission  environment  models  and  simulations,  a  more  complete 
understanding  of  total  system  performance  (people  plus  hardware  and  software  components)  and 
the  effects  of  mission  environment  factors  can  be  obtained.  This  set  of  slides  provides  an 
example  of  how  human  performance  modeling  could  be  applied  to  evaluate  potential  performance 
gains  achievable  by  incorporating  an  Unmanned  Aircraft  System  (UAS)  into  the  National  Security 
Cutter  (NSC)  system. 
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•  The  purpose  of  the  example  is  to  illustrate  how  human 
performance  modeling  could  be  applied  to  support 
evaluation  of  a  notional  UAS  use  concept 

•  Coast  Guard  concepts  for  UAS  use  are  still  evolving 

-  This  example  is  not  based  on  any  particular  UAS  concept  being 
considered 

-  The  HSI  team  did  not  want  to  distract  attention  from  the  purpose 
of  the  example  by,  perhaps,  misstating  a  current  concept 

•  Operational  inaccuracies  and  inconsistencies  might  exist 
in  the  example 

-  Please  forgive  and  disregard  these 
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Before  we  begin  with  the  discussion  of  the  example,  there  are  a  few  caveats  that  need  to  be 
made.  First,  this  example  is  completely  notional.  It  is  not  based  on  any  existing  UAS  concept 
(that  we  are  aware  of)  being  pursued  by  the  Coast  Guard.  While  the  Coast  Guard  is  actively 
pursuing  concepts  and  roles  for  UAS  capabilities,  those  concepts  are  still  evolving.  We  did  not 
want  to  distract  the  audience  from  the  message  of  our  example  by  misstating  a  concept.  Second, 
there  might  be  operational  inaccuracies  and  inconsistencies  in  the  notional  example  presented. 
Please  try  to  forgive  and  overlook  these.  Limited  time  and  resources  were  available  to  develop 
the  example  and  the  team  did  the  best  we  could. 
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The  UAS  Mission  with  the  NSC 


fi'pm  Science  to  ScMpns 


•The  primary  UAS  role 
would  be  to  detect, 
classify,  and  identify 
marine  contacts  within 
the  operational  area 
(OPAREA). 

•  Expected  performance 
improvements: 

•  Increased  likelihood  of 
vessel  detection 
•A  higher  percentage  of 
vessels  classified  and 
identified 
•Much  higher 
probability  that  targets 
of  interest  (TOI)  are 
identified  and 
interdicted. 
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This  slide  depicts  the  mission  context  of  our  notional  UAS-integrated-with-the-NSC  example.  In 
our  notional  example,  the  NSC’s  basic  mission  is  to  surveil  a  specified  operational  area 
(OPAREA);  detect,  classify,  and  identify  vessels  in  that  area;  identify  targets  of  interest  (TOI)  from 
the  complete  set  of  vessels  in  the  OPAREA;  and  prosecute  those  TOI  engaged  in  illegal  or 
dangerous  activity.  The  NSC  can  perform  this  mission  using  it’s  organic  surveillance  radar  to 
detect  and  classify  targets  but  the  radar’s  range  is  limited.  This  fact  coupled  with  the  NSC’s 
relatively  slow  speed  combines  to  constrain  the  area  that  can  be  surveiled  effectively. 

The  UAS  has  its  own  surveillance  radar  and  travels  at  much  higher  speeds  than  the  NSC. 
Consequently,  the  UAS  can  extend  significantly  the  size  of  the  OPAREA  that  can  be  covered. 

The  effect  on  mission  performance  should  be  to  increase  the  likelihood  that  vessels  in  the 
OPAREA  are  detected,  classified,  and  identified.  Equally,  the  likelihood  that  TOI  are  found  among 
the  detected  and  classified  vessels  should  be  increased.  Hence,  the  mission  effectiveness  of  the 
NSC  system  should  be  improved  significantly. 
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Operational  Concepts 


Fipra  Science  to  ScMons 


Baseline:  National  Security  Cutter  +  MH- 
65C 

-  Contact  Detection 

•  NSC  sails  surveillance  pattern 

•  NSC  organic  radar  detects  vessel(s) 

•  NSC  C2  center  evaluates  contacts  to  classify/ 
identify  to  extent  possible 

-  E.g.,  correlate  contact  transponder  data  with 
expected  traffic 

-  Contact  identification 

•  Multiple  unknown  contacts  prioritized  and 
plan  formulated  for  prosecuting  multiple 
intercepts 

•  NSC  intercepts  and  identifies  contacts  of 
interest 

•  MH-65C  launched  to  intercept  and  identify 
contacts  of  interest  that  NSC  cannot  overtake 

-  Contact  Management 

•  NSC  C2  center  maintains  contact/track 
history 


Test :  National  Security  Cutter  +  MH- 
65C  +  UAS 

-  Contact  Detection 

•  UAS  performs  primary  vessel  detection  role; 
NSC  is  secondary 

•  UAS  flies  planned  surveillance  pattern;  NSC 
sails  secondary  surveillance  pattern 

•  UAS  and  NSC  radars  detect  vessel(s) 

•  NSC  C2  center  evaluates  radar  contacts  to 
classify/  identify  to  extent  possible 

-  Contact  identification 

•  Multiple  unknown  contacts  prioritized  and 
UAS  flight  plan  is  formulated  for  prosecuting 
multiple  intercepts;  NSC  intercept  plan  for 
close  contacts 

•  UAS  intercepts  and  identifies  contacts  of 
interest 

•  NSC  intercepts  and  identifies  contacts  of 
interest 

•  MH-65C  launched  to  intercept  and  identify 
contacts  of  interest  that  NSC  cannot  overtake 

-  Contact  Management 

•  NSC  C2  center  maintains  contact/track 
history 
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In  order  to  evaluate  the  magnitude  of  any  effectiveness  gains  offered  by  UAS  technology, 
simulations  will  be  developed  that  represent  current  (baseline  condition)  NSC  capabilities  and  the 
addition  of  a  UAS  capability  (the  test  condition)  to  the  NSC  and  that  exercise  those  capabilities  in 
the  context  of  surveillance  missions  in  an  operational  environment.  The  NSC  and  UAS 
simulations  must  incorporate  behaviors  that  reflect  the  concepts  of  operations  (CONOPS)  that 
drive  system  employment.  This  slide  outlines  the  notional  CONOPS  associated  with  the  NSC  and 
UAS  capabilities. 

The  baseline  NSC  capability  consists  of  the  ship  itself  and  one  MH-65C  helicopter  that  operates 
off  the  ship.  Within  the  OPAREA  the  NSC  would  sail  a  surveillance  pattern  and  use  it’s  onboard 
radar  to  search  for  and  detect  vessels.  Radar  operators  in  the  NSC  Combat  Information  Center 
(CIC)  would  examine  the  radar  data  and  other  available  information  (e.g.,  Automated  Identification 
System  or  AIS  data)  to  classify  the  vessels.  Contacts  would  be  evaluated  to  determine  which  are 
possible  targets  of  interest.  In  instances  in  which  multiple  contacts  are  found  in  the  same  area, 
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personnel  in  the  CIC  will  prioritize  the  contacts  and  formulate  a  plan  for  prosecuting  them.  In 
some  instances  the  NSC  itself  will  intercept  a  TOI.  In  other  instances  the  MH-65C  will  be 
launched  to  intercept  and  identify  the  target.  Regardless  of  which  capability  is  used  to  make  the 
identification,  a  key  function  throughout  the  mission  will  be  to  keep  track  of  vessels  that  have  been 
detected,  classified,  and  identified  so  that  valuable  time  is  not  wasted  re-identifying  vessels  that 
have  been  previously  identified.  This  function  will  be  performed  in  the  CIC. 

The  test  condition  adds  a  UAS  to  the  NSC  plus  MH-65C  baseline  capability.  The  UAS  has  its  own 
surveillance  and  imaging  radar  and  electro-optical  (EO)  and  infrared  (IR)  sensor  capabilities.  The 
UAS  would  launch  from  the  NSC  and  use  its  surveillance  radar  and  speed  to  surveil  a  large 
portion  of  the  OPAREA.  The  UAS  surveillance  route  would  complement  the  route  sailed  by  the 
NSC  to  maximize  the  area  covered.  UAS  surveillance  radar  data  would  be  fed  to  the  NSC  CIC  to 
provide  a  common  operating  picture  of  contacts.  When  needed  the  UAS  imaging  radar  would  be 
used  to  provide  additional  target  classification  data.  As  in  the  baseline  condition,  CIC  personnel 
would  determine  potential  TOI  from  among  the  contacts  and,  in  the  case  of  multiple  contacts, 
prioritize  the  contacts  and  formulate  a  plan  for  intercepting  and  identifying  the  TOI.  The  UAS 
would  be  the  primary  means  for  accomplishing  TOI  identification  but  the  MH-65C  and  even  the 
NSC  could  be  used  to  make  identifications  in  instances  in  which  greater  efficiency  is  obtained  by 
not  diverting  the  UAS  from  its  current  activities.  As  in  the  baseline  condition,  a  key  CIC  function 
throughout  the  mission  will  be  to  keep  track  of  vessels  that  have  been  detected,  classified,  and 
identified. 
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Key  Human  Performance 
Elements  and  Issues 


Fium  Science  So  ScMpns 


•  Baseline:  National  Security  Cutter  +  MH- 
65C 

-  National  Security  Cutter 

•  Surveillance  pattern  planning 

•  Surveillance  pattern  execution 

•  Radar  scope  operation/  interpretation 

-  Contact  detection 

-  Data  interpretation 

-  Vigilance  effects 

-  Situation  awareness  (e.g.,  maintaining  contact 
history) 

-  MH-65C 

•  Intercept  management  (coordination  between 
helo  crew  and  NSC  radar  team) 

•  Contact  visual  detection  and  identification 
performance  of  aircrew 
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•  Test :  National  Security  Cutter  +  MH- 
65C  +  UAS 

-  UAS 

•  Surveillance/sensor  planning 

•  Surveillance  pattern  execution 

•  Contact  intercept  flight  planning 

•  Contact  intercept  execution 

•  Contact  visual  detection,  classification,  and 
identification  performance  of  aircrew 

-  National  Security  Cutter 

•  Surveillance  pattern  planning 

•  Surveillance  pattern  execution 

•  Radar  scope  operation/  interpretation 

-  Contact  detection 

-  Data  interpretation 

-  Vigilance  effects 

-  Situation  awareness  (e.g.,  maintaining  contact 
history) 

-  MH-65C 

•  Intercept  management  (coordination  between 
helo  crew  and  NSC  radar  team) 

•  Contact  visual  detection  and  identification 
performance  of  aircrew 
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Human  performance  modeling  will  be  one  of  the  simulation  technologies  applied  in  our  notional 
NSC-UAS  example.  A  first  step  to  developing  human  performance  models  is  to  identify  the 
human  behaviors  to  be  modeled.  This  slide  specifies  key  human  behaviors  to  be  modeled  in  the 
baseline  and  test  conditions.  It  is  important  to  understand  that  human  performance  modeling 
does  not  involve  detailed  modeling  of  all  human  activities  performed  in  a  system.  (Like  all 
hardware  and  software  elements  of  a  system  will  not  be  modeled  in  detail  in  a  system  simulation.) 
The  focus  of  human  performance  modeling  will  be  on  those  human-mediated  activities  that  are 
most  directly  related  to  the  mission  or  functions  of  interest.  In  our  notional  example  this  will  be  the 
personnel  and  activities  involved  in  conducting  the  surveillance  mission  and  detecting,  classifying 
and  identifying  targets  of  interest. 

In  the  baseline  condition  this  includes  bridge/navigation  personnel  and  their  development  and 
execution  of  the  surveillance  patterns  the  NSC  will  sail.  It  also  includes  the  Combat  Information 
Center  (CIC)  personnel  that  operate  and  monitor  the  NSC’s  surveillance  radar  and  data  systems 
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to  detect  and  classify  vessels,  identify  TOI,  and  maintain  contact  histones.  Given  that  the  MH-65C 
will  perform  most  of  the  target  intercepts  required  for  the  identification  process,  it  also  will  be 
important  to  model  the  aircrew’s  activities.  Executing  the  intercept  and  performing  a  visual 
identification  will  be  included  in  these. 

In  the  test  condition,  the  NSC  and  MH-65C  personnel  and  activities  described  above  would  be 
modeled  and  the  activities  of  the  UAS  crew  would  be  added.  UAS  crew  activities  would  include 
surveillance  pattern  planning  and  execution  (similar  to  the  surveillance  pattern  planning  required 
by  the  NSC),  contact  intercept  execution  (similar  to  that  performed  by  the  MH-65C),  and  visual 
identification  of  TOI  by  the  UAS  aircrew  (similar  to  the  MH-65C). 

Note  that  the  UAS  aircrew  shares  performances  that  are  similar  to  both  the  NSC  and  the  MH-65C. 
This  is  important  information  to  the  human  performance  modeler  because  it  suggests 
opportunities  to  reuse  model  elements  across  test  conditions.  Implementing  a  model  architecture 
that  supports  reuse  of  model  components  across  test  conditions  can  result  in  significant  savings  in 
the  cost,  time,  and  effort  required  to  build  the  human  performance  models.  Note  that  while  there 
are  similarities  in  the  tasks  and  activities  performed  by  different  teams,  detailed  elements  of  task 
performance  might  differ.  For  example,  the  algorithms  that  calculate  probability  of  target  detection 
and  identification  using  the  naked  eye  or  optical  aids  by  the  MH-65C  aircrew  might  use  minutes  of 
arc  subtended  by  the  target.  The  algorithms  that  calculate  probability  of  target  detection  and 
identification  using  the  UAS  camera  by  the  MH-65C  aircrew  might  use  pixels  subtended  on  the 
video  display  by  the  target.  Consequently,  even  when  model  components  are  reused  it  sometimes 
is  necessary  to  adapt  elements  to  fit  a  slightly  different  performance  requirement. 
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« i r 


JMIL. 

Fmfi  Science  to  Solutions 


OPAREA 
size  (nm2) 

Number  of 
vessels 
Dispersion 
of  vessels 

Vessel  size 

Vessel 

speed 

Sea  state 
Weather 
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A  particular  area  of  interest  in  our  study  will  be  how  factors  related  to  the  mission  environment 
affect  overall  mission  performance.  The  mission  being  performed  is  one  of  surveillance; 
detection,  classification,  and  identification  of  vessels  in  the  OPAREA;  and  identification  of 
particular  TOI.  Factors  in  the  environment  that  can  affect  the  mission  will  include  the  size  of  the 
OPAREA  to  be  surveiled;  the  number,  types,  and  sizes  of  vessels  to  be  detected  and  identified; 
the  speed  of  the  different  vessels;  the  extent  to  which  vessels  are  scattered  across  the  OPAREA; 
sea  states;  and  weather.  A  test  plan  would  be  developed  that  defines  test  conditions  made  up  of 
unique  combinations  of  levels  of  these  factors.  When  the  test  plan  is  executed  and  data  are 
collected,  statistical  and  other  analyses  will  use  the  mission  environment  factors  as  a  basis  for 
organizing  the  analysis,  characterizing  results,  and  extrapolating  results  into  environment  states 
not  tested. 
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System  Performance 
Parameters 


from  Science  to  Solutions 


ShipiwBrd/llAS  0  pc  rational  Performance  Parameters 

Operational 

B-ocmd’i 

Operational 

Expectations 

UAS  (single  flight  sortie  duration) 

5  Ins  (max) 

4  hrs 

UA-S  Oprraiing  distance  {from -cutter) 

10(1  nm  mas 

50  tun 

UAS  sortiefs^'day  {non-surge  ops) 

2  (max) 

1 

MI1-G5C  Sortics/day  with  iinglc  crew  (StuVflighi)  in  non-surg* 
operations 

3  (max) 

2 

Radar  range  to  detect  2  sq  meter  target  (in  sea  state  5) 

20  nm 

10  nm 

Cutter  Flight  Operations,  (uteiDde  ill  shifrboard  evolutions  & 
flight  limes) 

3.5  hrs 

3hrs 

BOAR  identification  range  for  TOT  (size) 

2  nra 

1  run 

Shipboard  aircraft  normal  daily  flight  hours  (U  AS  or  MH-65Q 

6htS  [mart) 

4  hrs 

Environmental  Availability  for  UAS  launch  and  recovery  to  CG 
Cutter 

60%  (mas) 

50% 

UAS  can  perform  the  Surveillance  and  Detection  Mission 

100%  (max) 

100%-  (mart) 

t)A$  can  perform  the  Classification  and  Identification  Mission 

75%  (tmx) 

50%  (rain) 

UAS  &  maimed  helicopter  flight  operations  will  only  be 
conducted  when  in  OP  AREA.  No  flights  while  the  cutter  is 
transiting  to/1  Pram  OPAREA. 
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In  our  notional  study  NSC,  MH-65C,  and  UAS  platform  hardware  and  software  functionality  will  be 
modeled  in  addition  to  human  performance.  This  is  necessary  to  represent  the  “total  system.” 
Modeling  the  platforms  also  is  important  as  a  means  of  establishing  bounds  for  human 
performance  (e.g.,  range  and  resolution  of  the  UAS  optical  sensor).  This  slide  lists  some  system 
performance  parameters  that  were  specified  for  a  previous  USCG  UAS  study.  Note  that 
parameters  are  not  always  physical  factors.  They  can  be  operational  factors  as  well.  Factors  like 
sorties  per  day  and  sortie  duration  also  are  important  because  they  establish  operational  bounds 
on  UAS  performance  that  ultimately  affect  mission  factors  such  as  the  size  of  the  area  that  can  be 
surveiled  in  a  twenty-four  hour  period. 
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Mission 


System 

Function 


Performance  Metrics 


Fjpra  Science  to  Solutions 


•%  TOI  detected 
•%  TOI  identified 


Operator 
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Contact  Detection 

-  %  OPAREA  surveiled 

-  Avg  interval  between 
repeated  surveillance 
events 

-  %  Vessels  detected 

-  Avg  time  to  detect 

Contact  Identification 

-  %  Vessels  classified 
correctly 

-  %  Vessels  identified 
correctly 

1  1 

•  %  AIS  contacts  detected 

•  Avg  time  to  detect  AIS 
contacts 

•  %  skin  paint  contacts 
detected 

•  Avg  time  to  detect  skin 
paint  contacts 

•  Avg.  time  to  classify 
vessels 

•  Avg.  time  to  ID  vessels 

•  Avg.  range  at 
classification 

•  Average  range  at 
identification 

Contact  Mgt. 

-  Avg  detects  and 
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Ultimately,  the  purpose  of  modeling  and  simulation  is  to  produce  data  that  informs  acquisition 
decision-makers  about  factors  related  to  an  acquisition.  Typically,  these  data  provide  measures  of 
simulation  outcomes.  Outcomes  usually  are  measured  at  multiple  levels  of  system 
decomposition.  Mission  performance  measures  are  the  highest  level  of  assessment.  Mission 
performance  measures  provide  a  sort  of  bottom-line  for  comparing  overall  performance  among  the 
alternatives.  Performance  measurement  schemes  also  include  other  lower  level  measures  that 
assess  operation  of  key  mission  functions  and  system  components  and  elements.  These 
measures  provide  an  in-depth  understanding  of  how  lower  level  system  capabilities  contribute  to 
(or  detract  from)  overall  mission  performance. 


This  slide  offers  some  performance  measures  that  could  be  used  in  our  notional  UAS  study.  At 
the  mission  level  measures  are  focused  on  assessing  the  essential  outcomes  of  the  surveillance 
mission.  In  the  context  of  our  notional  study  the  mission  of  the  NSC  system  is  to  find  and  correctly 
identify  targets  of  interest.  Mission  performance  is  perfect  when  TOI  detections  and  identifications 
are  100%. 


Acquisition  Directorate 

Research  &  Development  Center 


F-l  1 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


At  the  system  level  there  are  enabling  functions  that  must  be  accomplished  if  the  mission  is  to  be 
successful.  The  first  is  to  simply  detect  vessels  in  the  OPAREA.  The  ability  to  detect  vessels  is 
directly  related  to  the  ability  to  surveil  the  entire  OPAREA.  Given  that  the  OPAREA  probably  is 
larger  than  the  area  that  can  be  surveiled  by  a  sensor  from  one  location  in  the  OPAREA,  a 
surveillance  pattern  must  be  established  that  allows  the  OPAREA  to  be  covered.  When  OPAREA 
size  is  combined  with  sensor  “footprint”  (the  area  a  surveillance  sensor  can  cover)  and 
surveillance  platform  speed,  another  important  factor  that  emerges  is  how  often  segments  of  the 
OPAREA  are  surveiled.  This  is  important  because  vessels  move  through  the  OPAREA  over  time. 
Ideally,  surveillance  intervals  for  portions  of  the  OPAREA  must  be  sufficiently  short  to  ensure  that 
a  vessel  crossing  through  the  OPAREA  is  detected.  The  %  of  vessels  detected  provides  an 
overall  evaluation  of  the  detection  function.  The  average  time  to  detect  vessels  entering  the 
OPAREA  is  a  measure  of  the  efficiency  of  the  detection  process.  Vessel  identification  is  another 
key  enabling  function.  There  are  two  steps  in  the  identification  process.  The  first  is  to  classify  a 
vessel  (e.g.,  type,  size).  The  second  is  to  identify  it  (e.g.,  registration,  name,  country  of  origin). 
Errors  in  either  step  can  lead  to  TOI  not  being  identified  correctly.  Finally,  contact  management  is 
the  process  of  keeping  track  of  vessels  that  have  been  detected,  classified,  and  identified  so  that 
valuable  time  is  not  spent  performing  that  process  yet  again.  The  average  number  of  detections 
and  classifications  of  a  vessel  is  a  measure  that  recognizes  that  a  vessel  might  be  detected 
multiple  times  by  sensors  as  it  crosses  the  OPAREA  and  the  NSC  conducts  its  surveillance  route. 
The  detection  event  will  require  some  classification  activity  for  the  NSC  crew  to  deduce  that  it  is  a 
previously  identified  vessel.  The  measure  of  contact  management  efficiency  is  the  %  of 
previously  identified  contacts  that  are  re-identified.  The  larger  the  number  the  more  time  that  is 
being  wasted  on  re-identifications. 


People  perform  critical  roles  in  the  contact  detection,  identification,  and  management  processes 
and  some  possible  human  performance  measures  are  offered  in  the  slide.  Visual  observation  of 
radar  screens  is  described  as  a  vigilance  task  because  operator  must  be  “vigilant”  to  detect  the 
appearance  of  a  target  symbol  at  any  time  on  any  portion  of  the  display.  In  our  notional  study  the 
probability  that  a  NSC  radar  operator  detects  radar  contacts  of  vessels  would  be  determined 
jointly  by  factors  such  as  the  assumed  visual  capabilities  of  the  operator,  the  duration  of  the 
contact  symbol  on  the  display,  the  proximity  of  the  contact  to  other  contacts,  and  the  number  of 
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times  the  vessel  comes  within  the  NSC’s  surveillance  pattern  over  the  course  of  a  mission.  Thus, 
the  human  performance  outcome  is  only  partially  deterministic,  based  on  HPM  parameters.  The 
actual  performance  observed  is  the  confluence  of  a  number  of  “real  world”  factors.  In  our  notional 
example  there  are  two  different  types  of  contact  that  operators  would  detect.  The  first  is 
Automatic  Detection  System  (AIS)  contacts.  AIS  uses  radio  broadcast  of  digital  data  about  a 
vessel  to  provide  course,  speed,  and  other  data  about  that  vessel  to  other  vessels  within  radio 
range.  AIS  data  are  displayed  with  symbols  and  labels  that  are  expected  to  be  more  readily 
detected  than  the  other  category  of  contacts,  which  is  “skin  paints.”  Skin  paints  are  radar 
generated  symbols  that  indicate  objects  the  radar  has  detected.  Two  measures  are  used  for  each 
contact  type.  One  is  the  %  of  contacts  detected  by  the  operator.  The  other  is  the  average  time 
required  to  detect  a  contact  once  it  is  displayed  on  the  radar.  Operator  classification  and 
identification  performance  is  captured  by  measures  of  time  and  range.  Lower  times  mean  more 
efficient  performance.  Conversely,  longer  ranges  also  mean  more  efficient  performance  because 
less  time  is  needed  to  get  close  enough  to  make  a  classification  or  identification  decision.  Finally, 
contact  management  is  measured  in  terms  of  errors  that  operators  can  make  in  determining 
whether  a  detected  vessel  is  one  previously  identified  or  is  a  new  vessel  that  has  not  been 
identified.  Instances  in  which  a  new  contact  is  misidentified  as  an  old  one  means  that  an 
identification  process  might  not  be  performed.  This  makes  it  possible  to  overlook  a  TOI. 

Instances  in  which  old  contacts  are  misidentified  as  a  new  one  means  that  time  will  be  wasted  re¬ 
identifying  a  vessel. 
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In  our  notional  study,  a  simulation  environment  is  envisioned  that  consists  of  three  major 
components.  The  mission  environment  simulation  would  provide  entities  that  represent  the  TOI 
and  other  vessel  traffic  in  the  OPAREA.  It  also  would  provide  the  capability  to  simulate  sea 
states,  weather,  and  other  environmental  factors  that  can  affect  mission  performance.  Multiple 
simulations  might  be  used  to  represent  these  elements  of  the  mission  environment. 

The  mission  entities  component  of  the  testbed  would  provide  the  simulations  of  the  NSC,  MH- 
65C,  and  UAS  platforms  and  sensors.  Attributes  of  the  platforms  that  would  be  modeled  include 
cruising  and  intercept  speeds,  operating  altitudes,  waypoint  navigation  (following  a  preplanned 
surveillance  or  other  route  through  a  series  of  waypoints),  and  “task-ability.”  Task-ability  is  a 
particularly  important  feature.  It  means  the  ability  to  receive  and  respond  to  directions  to  change 
course  and  speed  to  perform  maneuvers  such  as  intercepting  a  vessel  so  it  can  be  identified. 
One  of  the  functions  of  the  different  NSC  platform  crews  modeled  in  the  HPM  will  be  to  make 
decisions  regarding  which  vessels  to  intercept  and  when.  The  different  NSC  platform  entities 


Acquisition  Directorate 

Research  &  Development  Center 


F-14 


Unclassified  |  RDC  |  C.  Hale,  et  al.  |  Public  |  April  2009 


Survey  of  HSI  Tools  for  IISCG  Acquisitions 


must  be  able  to  implement  these  decisions.  Because  surveillance  is  such  a  significant  portion  of 
the  mission  in  our  notional  study,  modeling  the  sensors  used  by  the  different  NSC  platforms  is 
essential.  This  includes  representing  both  the  types  of  sensors  (radar,  EO,  and  IR)  and  their 
modes  (e.g.,  moving  target  indicator  mode  or  MTI  and  imaging  for  the  radars).  We  should  point 
out  that  sensor  modeling  in  the  context  of  this  example  would  not  include  generating  actual 
images.  Rather,  it  would  consist  of  generating  data  about  an  image  or  other  output  of  the  sensor. 
The  model  for  an  EO  sensor,  for  example,  might  combine  data  on  target  size,  shape,  orientation, 
and  range  with  information  on  mode  and  resolution  of  the  sensor  to  estimate  the  number  of  pixels 
the  target  would  occupy  on  the  EO  display.  The  HPM  would  use  this  data  to  determine  whether 
and-or  when  the  target  is  detected  or  identified. 

The  human  performance  model  component  would  represent  the  NSC  platform  personnel 
performing  key  mission  activities.  These  would  include  personnel  that  man  the  CIC  and  bridge  on 
the  cutter  as  well  as  the  UAS  pilot  and  sensor  operator,  and  the  MH-65  aircrew.  Specific 
performances  that  would  be  modeled  are  presented  on  the  next  slide. 
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While  the  NSC  is  a  large  vessel  with  a  sizeable  crew,  only  are  small  number  of  personnel  will  be 
modeled  in  our  notional  study.  This  reflects  a  fundamental  principle  of  modeling  and  simulation, 
which  is  to  focus  modeling  efforts  on  those  factors  that  really  matter.  For  the  purposes  of  our 
study  we  assume  that  if  everyone  else  in  the  NSC  system  does  their  jobs  correctly  then  it  is  the 
personnel  listed  on  this  slide  that  will  affect  the  outcome  of  the  surveillance  mission.  Among  the 
standard  personnel  complement  of  the  cutter  it  is  personnel  from  the  CIC  and  bridge  personnel 
that  are  most  involved  in  mission  execution.  The  behaviors  that  we  would  model  for  CIC 
personnel  consists  of  monitoring  the  NSC  radar  display  to  detect  contacts,  evaluate  and  classify 
those  contacts,  determine  which  contacts  are  possible  TOI  and  require  intercept,  coordinate  target 
intercept  with  the  platform  making  the  intercept,  and  maintaining  a  contact  history  in  an  effort  to 
avoid  re-identifying  previously  identified  targets.  Modeling  of  the  cutter  bridge  personnel  is  limited 
to  planning  and  executing  intercepts  (when  the  cutter  is  making  the  intercept).  Note  that 
surveillance  route  planning,  which  was  listed  as  a  key  mission  behavior  earlier  in  the  briefing,  is 
not  listed  as  a  modeled  behavior.  This  is  because  the  surveillance  route  or  pattern  would  be 
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determined  prior  to  mission  execution  and  would  be  part  of  the  platform  script.  Consequently,  the 
HPM  would  not  be  generating  these  routes. 

Both  UAS  pilot  and  sensor  operator  performance  would  modeled.  Key  pilot  behaviors  that  would 
be  modeled  include  take-offs  and  landing  from  the  NSC;  determining  and  entering  waypoints 
based  on  planned  routes  and  intercept  routes  (in  many  UAS  pilots  do  not  actually  manipulate  stick 
and  throttle  controls;  they  enter  waypoint  data  into  a  mission  computer  and  automation  software 
flies  the  UAS  to  the  waypoint);  and  managing  the  positioning  of  the  UAS  so  that  a  target  remains 
in  the  sensor  view  or  a  target  intercept  is  achieved.  The  sensor  operator  model  would  need  to 
coordinate  UAS  radar  surveillance  with  the  CIC.  Remember,  the  CONOPS  requires  integrating 
the  cutter  and  UAS  radars  to  maximize  OPAREA  coverage.  This  requires  some  coordination 
between  CIC  radar  operators  and  the  UAS,  particularly  when  there  is  overlapping  coverage  and 
common  targets  must  be  resolved.  The  sensor  operator  also  must  employ  the  EO  and-or  IR 
sensors  to  identify  targets.  Finally  and  as  part  of  the  intercept  and  the  identification  processes, 
the  sensor  operator  must  coordinate  with  the  pilot  to  ensure  that  the  UAS  is  positioned  such  that 
the  sensor  operator  keeps  the  target  in  view  and  obtains  good  imagery  of  the  target. 

The  most  limited  portion  of  the  human  performance  modeling  would  be  with  the  MH-65C  aircrew. 
Here  we  would  model  take-offs  and  landings  from  the  NSC;  coordinating  target  intercepts  with 
CIC;  and  visually  acquiring,  classifying,  and  identifying  vessels. 
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To  create  an  actual  human  perform  model,  the  behavioral  descriptions  provided  in  the  previous 
slide  would  be  transformed  into  HPM  elements  within  a  human  performance  modeling 
environment.  This  slide  shows  a  human  performance  model  constructed  using  the  Army 
Research  laboratory’s  Improved  Performance  Research  Integration  Tool  (IMPRINT)  environment. 
IMPRINT  uses  task  network  modeling  to  represent  human  performance.  As  the  name  implies, 
task  networks  use  a  flow  chart  type  format  to  specify  detailed  task  performance  sequences, 
decision  points,  and  branches.  Task  networks  are  organized  in  terms  of  higher-level  functions. 
Functions  are  organized  in  terms  of  goal  states.  Goals  states  are  a  concept  from  cognitive 
psychology.  The  idea  is  that  goals  provide  an  executive  function  that  organizes  and  controls 
behavior.  Goals  are  triggered  by  conditions  in  the  mission  environment  that  the  performer  needs 
to  maintain  within  certain  limits.  For  example,  a  navigation  goal  state  would  be  triggered  in  a  pilot 
when  he/she  determines  the  aircraft  is  off  course.  Once  the  goal  state  triggers,  lower-level 
functions  and  tasks  would  be  activated  to  bring  the  airplane  back  on  course  (restore  performance 
within  desired  limits). 
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Modeling  environments  such  as  IMPRINT  generally  have  two  major  components.  The  first  is  a 
model  development  module.  In  IMPRINT  this  is  a  graphical  interface  in  which  the  user  specifies 
goals,  functions,  and  tasks  and  arranges  their  organization  and  sequences.  The  user  also  enters 
data  that  controls  how  the  goals,  functions,  and  tasks  execute.  These  data  include  trigger  criteria, 
task  performance  times  and  variances,  task  output  and  effects  data,  and  behavioral  process 
algorithms  (e.g.,  visual  detection  and  identification,  information  processing,  etc.).  The  second 
component  is  a  runtime  module.  This  module  actually  executes  the  model,  manages  time, 
generates  and  collects  data,  and  manages  any  interfaces  to  other  models  and  simulations.  Task 
networks  similar  to  those  shown  in  the  slide  would  be  developed  for  each  of  the  NSC  system 
personnel  modeled  in  our  notional  example. 
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Defining  model  elements  is  only  one  part  of  the  model  development  process.  Once  tasks  have 
been  specified,  data  must  be  provided  that  governs  how  the  task  executes.  The  simplest 
constituent  of  task  data  is  task  performance  time.  This  slide  shows  an  IMPRINT  data  screen  for 
entering  task  performance  time  data  for  a  task.  Other  more  complex  data  might  be  entered  for 
tasks  also.  These  data  often  are  entered  in  “effects”  fields  provided  by  other  IMPRINT  screens. 
Effects  data  are  used  to  describe  detailed  task  performance  and  capture  the  output  of  that 
performance.  Effects  data  can  be  entered  as  expressions,  algorithms,  and  even  computer  code, 
Often  effects  data  are  used  to  model  behavioral  processes  involved  in  a  task.  In  our  notional 
example  these  would  include  modeling  visual  perception  of  vessels,  and  remembering  vessels 
that  have  been  identified. 
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•  Vigilance 

-  A  factor  in  sustained  attention  tasks  that  typically  require  observers  to  monitor 
displays  over  extended  periods.  The  likelihood  of  detecting  events  of  interest 
varies  as  a  function  of  time,  event  density,  and  attributes  of  the  displayed  event 
data. 

-  In  this  study  the  primary  effects  would  be  on  detection  of  radar  contacts. 

•  Fatigue 

-  A  complex  state  affected  by  factors  such  as  how  long  an  individual  has  been 
awake,  the  duration  of  recent  sleep,  where  an  individual  is  in  his  or  her  circadian 
cycle  of  alertness;  physical,  cognitive,  and  or  attentional  demands  of  the  job. 

-  In  this  study  the  primary  effects  would  be  on  detection  of  radar  contacts  (a 
potential  interaction  with  vigilance)  as  well  as  the  time  and  accuracy  of  acquiring, 
classifying,  and  identifying  vessels  visually  and  with  the  UAS  EO/IR  capability. 

•  Memory 

-  The  ability  to  retain  and  recall  information  quickly  and  accurately. 

-  In  this  study  the  primary  effects  would  be  on  determining  which  contacts  have 
been  identified  already  so  that  time  is  not  wasted  re-identifying  them. 
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In  the  previous  slides  we  have  discussed  modeling  in  terms  of  representing  performance  and 
effects  at  the  level  of  tasks.  But,  there  are  behavioral  processes  that  operate  over  time  and  can 
affect  performance  across  tasks.  These  often  are  referred  to  as  performance  moderators.  Within 
the  context  of  our  notional  example  there  are  several  that  should  be  considered  as  part  of  the 
human  performance  modeling  effort. 

Vigilance  is  a  factor  in  sustained  attention  tasks  that  typically  require  observers  to  monitor  displays 
over  extended  periods.  The  likelihood  of  detecting  events  of  interest  varies  as  a  function  of  time, 
event  density,  and  attributes  of  the  displayed  event  data.  In  our  notional  study  the  primary  effects 
would  be  on  detection  of  radar  contacts.  Fatigue  is  a  complex  state  affected  by  factors  such  as 
how  long  an  individual  has  been  awake,  the  duration  of  recent  sleep,  where  an  individual  is  in  his 
or  her  circadian  cycle  of  alertness;  physical,  cognitive,  and-or  attentional  demands  of  the  job.  In 
this  study  the  primary  effects  would  be  on  detection  of  radar  contacts  (a  potential  interaction  with 
vigilance)  as  well  as  the  time  and  accuracy  of  acquiring,  classifying,  and  identifying  vessels 
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visually  and  with  the  UAS  EO/IR  capability.  Memory  is  the  ability  to  retain  and  recall  information 
quickly  and  accurately.  In  this  study  the  primary  effects  would  be  on  determining  which  contacts 
have  been  identified  already  so  that  time  is  not  wasted  re-identifying  them. 
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Data  Sources  for  HPM 
Parameters 


fi'pra  Science  to  ScMons 


•  Human  performance  literature  and 
models 

-  Performance  times,  accuracies, 
error  rates 

-  Vigilance  effects 

-  Fatigue  effects 

-  Memory  performance 

•  UAS  Operations  Specific  Data 

-  UAS  context  specific  performance 
attributes/  effects 

-  Air  Force  Predator  Operations 
Centers 

-  Army  and  Navy  UAV  experts 

•  USCG  Subject  Matter  Experts  for 
USCG  Platforms  and  Operations 

-  UAS  context  specific  performance 
attributes/  effects 

-  CIC  and  radar  operations 

-  MFI-65C  aircrews 


•  Platform  Specifications/ Technical 
Data 

-  Fire  Scout  UAS  flight  operations 
specifications/  profiles 

-  Fire  Scout  UAS  EO/IR  sensor 
package  specifications 

-  Telephonies  RDR-1700B  radar 
specifications 

-  NSC  platform  specifications 

-  NSC  radar  specifications 

-  MFI-65C  flight  operations 
specifications/  profiles 

-  MH-65C  sensor  specifications 
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A  particularly  important  part  of  the  human  performance  modeling  process  is  obtaining  the  data 
need  to  set  parameters  within  the  HPM  and  provide  the  algorithms  for  the  behavioral  processes 
that  need  to  be  modeled.  The  data  used  can  significantly  affect  the  validity  of  the  HPM  that  is 
produced.  This  slide  presents  some  possible  data  sources  and  types  that  could  be  used  in  our 
notional  scenario. 

The  first  source  is  the  behavioral  science  literature  on  human  performance.  This  can  be  a  rich 
source  of  information,  particularly  for  fairly  common  behaviors  (e.g.,  visual  detection  of  different 
objects  and  data)  and  behavioral  processes  (e.g.,  fatigue,  memory  effects).  Often,  however,  other 
sources  are  needed  to  gain  insight  into  the  platform  or  operational-specific  performances  and  data 
required  by  a  model.  Operational  and  platform-specific  documents  on  systems,  operations, 
doctrine  and  tactics  and  subject  matter  experts  (SME)  are  the  sources  of  this  information.  We 
would  anticipate  using  two  operational  data  sources  for  our  notional  study.  One  would  be  defense 
organizations  that  currently  are  using  UAS  operationally.  This  includes  the  Air  Force,  Navy,  and 
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Army.  Because  UAS  are  new  to  the  USCG,  there  is  no  experience  base  from  which  to  obtain 
operationally-related  UAS  data.  By  interacting  with  experts  from  the  other  services  we  can  obtain 
data  on  UAS  context  specific  performance  attributes  and  effects  that  we  can  incorporate  into  our 
models  and  simulations.  This  will  help  us  model  UAS  processes  and  tactics  more  realistically. 
Similarly,  it  is  important  to  accurately  model  processes  and  activities  of  CIC  and  MH-65C 
personnel.  USCG  SME  would  be  used  to  obtain  this  data.  Finally,  it  will  be  important  to  have 
accurate  operational  data  for  the  NSC  platforms  and  sensors  that  the  HPM  employs  to  conduct 
the  surveillance  mission.  These  system  components  provide  capabilities  that  both  enable  and 
constrain  mission  performance  by  NSC  personnel.  This  slide  lists  some  of  the  platform  specific 
data  that  would  be  of  interest.  The  Fire  Scout  UAS  is  used  here  as  an  example  UAS.  The  Fire 
Scout  is  currently  in  use  by  the  Navy  and  is  one  of  the  UAS  being  considered  by  the  USCG. 
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Notional  Results 


•  Mission  performance  of 
NSC  system  with  UAV . 
is  substantially  better 

•  Key  UAS  performance 

factor:  UAS  speed, 
range,  sensor  allows 
more  frequent  — - 

“sampling”  of  same 
OPSAREA  segments 
for  mission-objective 
vessels 

•  Frequent  re-sampling 
increases  likelihood  a  - 
vessel  will  fall  within  a 
sensor  footprint.  Also, 
provides  multiple  looks 
at  a  vessel 

•  Operators  make  some 
errors  but  these  are  - 
offset  by  multiple 
opportunities  to 
evaluate/  reevaluate  the 
same  vessels 
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This  slide  presents  some  very  notional  results  that  might  be  obtained  from  our  notional  study.  The 
intention  of  this  slide  is  to  demonstrate  how  simulation  data  can  provide  insight  to  the  performance 
of  system  alternatives  and  where  and  why  differences  occur.  Not  unexpectedly,  the  data  show 
that  the  UAS  enabled  NSC  out  performs  the  baseline  NSC  significantly.  The  lower  level  data 
explain  why. 

The  main  reason  is  that  the  UAS  does  a  much  better  job  surveiling  the  OPAREA.  A  score  of 
825%  on  the  “%  OPAREA  Surveiled”  measure  means  that  the  UAS  covered  the  entire  OPAREA 
eight  times  over  the  duration  of  the  mission.  (Remember  the  OPAREA  is  larger  than  the  sensor 
footprint  of  the  UAS  so  the  UAS  must  fly  a  surveillance  pattern  to  cover  the  OPAREA.)  This 
means  that  portions  of  the  OPAREA  are  revisited  more  frequently  by  the  UAS  than  the  cutter, 
which  only  covered  200%  of  the  OPAREA.  The  result  is  that  the  UAS  has  more  opportunities  to 
detect  vessels  passing  through  the  OPAREA.  This  is  demonstrated  in  the  “Average  Number  of 
Detection  Opportunities  per  Vessel”  metric.  The  ultimate  effect  is  seen  in  the  “%  Vessels 
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Detected”  metric:  100%  for  the  UAS  versus  65%  for  the  cutter  alone.  Interestingly,  a  side  effect 
of  the  UAS’s  more  effective  coverage  of  the  OPAREA  is  the  greater  number  of  “Average  Detects 
and  Classifies  per  Vessel.”  This  suggests  that  the  contact  management  job  probably  is  more 
difficult  when  using  the  UAS  because  targets  are  revisited  more  often  and  there  are  more 
opportunities  to  get  confused  and  re-identify  an  old  target. 

Finally  and  as  regards  operator  performance,  there  is  relatively  little  difference  between  contact 
detection  and  identification  metrics.  To  some  extent  this  is  not  surprising  given  the  nature  of  the 
tasks  are  similar  across  the  different  systems.  The  most  remarkable  result  is  the  interaction  of  the 
“%  Correct  IDs”  metric  with  the  operator  contact  management  metrics  and  the  system  level 
contact  management  metrics.  Overall  the  operator  model  correctly  identified  vessels  only  93%  of 
the  time  but  at  the  mission  level  97%  of  the  TOIs  were  identified  correctly.  This  difference  is  due 
to  the  fact  that  because  the  UAS  revisited  targets  more  often  than  the  cutter,  the  UAS  sensor 
operator  had  more  opportunities  to  correct  wrong  IDs.  This  ultimately  led  to  more  effective  overall 
mission  performance. 
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Beyond  the  Numbers 


fi'pra  Science  to  ScMons 


•  Providing  a  quantitative,  operational  basis  for  comparing  system 
alternatives  is  only  one  benefit  of  a  NSC  +  UAS  modeling  and 
simulation  testbed 

•  Other  uses: 

-  Exploring  performance  bounds 

•  Interaction  of  mission  duration  with  vigilance  and  fatigue 

-  Affects  crew  size,  operational  tempo,  etc. 

•  Interaction  of  vessel  density  with  ability  to  effectively  manage  contact  history 

-  Affects  crew  size  also;  implications  for  CIC  track  management  tools 

-  Developing  tactics 

•  Surveillance  route  patterns 

-  Cooperative  patterns  to  maximize  surveillance  areas 

-  How  best  to  use  MH-65C  for  surveillance,  if  at  all 

•  Integrated  sensor  use 

-  How  best  to  integrate  NSC  and  UAS  radars  into  a  single  contact  detection  and 
tracking  capability 

-  Contact  types  for  which  radar  imagery  is  suitable  for  vessel  identification 

»  Contact  types  for  which  EO/IR  imagery  is  necessary  for  vessel  identification 

-  How  best  to  integrate  AIS  for  cross-cuing  UAS  sensors  for  target  identification 
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While  most  of  this  presentation  has  focused  on  modeling  and  simulation  as  a  means  to  generate 
data  for  acquisition  decision-making  there  are  other  benefits  that  should  be  highlighted  as  well. 
Some  of  these  are  listed  on  this  slide.  Exploring  performance  bounds  is  accomplished  by 
adjusting  system  or  mission  environment  parameters  until  mission  performance  becomes 
unacceptable.  For  example,  this  might  include  adjusting  OPAREA  size  or  number  of  vessels  in 
the  area  beyond  expected  limits  until  the  mission  fails.  This  will  provide  insight  into  the  robustness 
of  the  system  and  its  ultimate  capacity.  Exploring  boundaries  also  could  involve  manipulating 
factors  such  as  mission  duration  and  number  of  vessels  surveiled  to  obtain  a  better  understand  of 
possible  human  performance  limits.  This  information  can  help  resolve  HSI  issues  such  as  crew 
size,  watch  length,  sorties  per  day,  etc. 

Another  powerful  application  of  modeling  and  simulation  is  the  development  of  tactics  for  a  new 
systems.  Often,  system  developers  don’t  have  a  complete  understanding  of  the  ways  to  employ 
new  technologies  and  capabilities  that  maximize  mission  performance.  Simulation  provides  a 
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means  of  experimenting  with  different  concepts,  processes,  and  tactics  and  objectively  measuring 
the  outcomes  to  determine  what  works  best.  Our  notional  testbed  described  in  the  example  could 
be  used  to  address  issues  such  as  the  most  effective  cooperative  surveillance  patterns  for  the 
UAS  and  cutter  that  maximize  OPAREA  coverage  and  how  best  to  incorporate  the  MH-65C  for 
surveillance  (if  at  all).  Sensor  use  would  be  another  potential  area  of  study.  Issues  addressed 
here  might  include:  how  best  to  integrate  NSC  and  UAS  radars  into  a  single  contact  detection  and 
tracking  capability;  the  contact  types  for  which  radar  imagery  is  suitable  for  vessel  identification 
and  the  contact  types  for  which  EO/IR  imagery  is  necessary  for  vessel  identification;  and  how  best 
to  integrate  AIS  for  cross-cuing  UAS  sensors  for  target  identification. 

While  there  would  be  specific  test  conditions  that  would  be  constructed  to  explore  these  different 
issues  and  performance  data  would  be  obtained,  the  knowledge  gained  exceeds  the  numbers  that 
result.  The  ultimate  benefit  is  stronger,  more  effective  system  concepts  and  a  better 
understanding  of  how  to  maximize  system  performance  capabilities  and  avoid  limitations. 
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